OpenLineage

OpenLineage is an open standard for collecting data lineage. This guide explains how to send OpenLineage RunEvent, JobEvent, and DatasetEvent data to Guance from Airflow, Spark, dbt, or an application using an OpenLineage client.

Prerequisites¶

Before you begin, prepare the following:

• The data ingestion URL for your site, such as https://openway.guance.com. Use the actual URL for other sites or private deployments.
• The token for the target workspace. Treat the token as a secret; do not hard-code it or commit it to version control.
• A data job or client that supports the OpenLineage HTTP transport.
• HTTPS connectivity from the job runtime to the ingestion URL.

The examples use these placeholders:

Ingestion Protocol¶

The complete URL is:

<OPENLINEAGE_BASE_URL>/v1/write/openlineage

Each request accepts one event. JSON arrays and the OpenLineage batch endpoint are not supported. Keep each event well below the limit so that large facets do not interfere with delivery or retries.

Authentication¶

The endpoint supports these authentication methods:

If a request includes more than one token, the precedence is query token, X-Token, then Bearer. Do not configure different tokens in the same request.

Event Requirements¶

Events must follow the OpenLineage specification. All three types require eventTime (an ISO 8601 timestamp), producer (a producer URI), and schemaURL (the specification version URL). Guance identifies the type from the top-level run, job, and dataset objects:

Include inputs and outputs when a RunEvent or JobEvent has input or output datasets; each dataset should have a namespace and name. Events for the same run must keep the same runId, job.namespace, and job.name. A run normally emits START followed by a terminal COMPLETE, FAIL, or ABORT.

Event Sources and Record Types¶

After ingestion, source identifies the original OpenLineage event type; lineage_record_type identifies the record Guance projected from that event. These are different dimensions: one RunEvent can produce multiple records, all with source=run_event.

An event record stores the complete, redacted event JSON in lineage_message. job and dataset are entity records; dataset_edge links a Job/Run to an input or output Dataset; schema_version stores a schema summary; and field_edge represents column-level lineage. Without relevant inputs, outputs, or facets, the corresponding derived records are not created. Rejected-event audit records may use source=unknown_event and lineage_record_type=ingest_error; this is not a fourth supported input type.

Guance reserves source for event classification. If the original top-level JSON contains a string source, projected records preserve it as src_source instead of replacing Guance's source. It also remains in the event record's lineage_message.

Queryable Field Inventory¶

The following are top-level fields extracted by Guance, not every field in the original OpenLineage JSON. Presence depends on the event type, record type, and available facets; an absent field is not necessarily an empty value. For the complete event, inspect lineage_message on a lineage_record_type=event record (sensitive values may be redacted). Use event_id to relate records projected from the same event; run_id, job_id, and dataset_id relate runs, jobs, and datasets respectively.

Common Fields and Event Payload¶

Run and Job Fields¶

Identity fields propagate to records derived from the same event. Job names, state, and facet-extracted fields are mainly on event; Job metadata from a JobEvent is also on the job record.

Dataset, Schema, and Lineage Edge Fields¶

Only DatasetEvent produces a dataset record. RunEvent and JobEvent produce dataset_edge records from their inputs and outputs. schema_version holds a schema summary; the complete schema remains in the original facet inside lineage_message.

ingest_error records additionally have ingest_error_reason_code (rejection reason), ingest_error_payload_hash (payload digest), and ingest_error_payload_size (payload size in bytes). They never store the rejected raw payload.

Quick Verification¶

Save the following as event.json, and generate a new runId for each test:

{
  "eventType": "START",
  "eventTime": "2026-10-21T08:00:00Z",
  "run": {
    "runId": "4a8f90a7-79c0-4f80-89e7-6b6c5d4fc79a"
  },
  "job": {
    "namespace": "example.scheduler",
    "name": "daily_orders"
  },
  "inputs": [
    {
      "namespace": "postgresql://warehouse.example:5432",
      "name": "sales.orders"
    }
  ],
  "outputs": [
    {
      "namespace": "postgresql://warehouse.example:5432",
      "name": "analytics.daily_orders"
    }
  ],
  "producer": "https://example.com/openlineage/integration",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent"
}

Send the event with a Bearer token:

curl --fail-with-body --silent --show-error \
  -X POST '<OPENLINEAGE_BASE_URL>/v1/write/openlineage' \
  -H 'Authorization: Bearer <WORKSPACE_TOKEN>' \
  -H 'Content-Type: application/json' \
  --data-binary @event.json

Send a gzip-compressed event:

gzip -c event.json | curl --fail-with-body --silent --show-error \
  -X POST '<OPENLINEAGE_BASE_URL>/v1/write/openlineage' \
  -H 'Authorization: Bearer <WORKSPACE_TOKEN>' \
  -H 'Content-Type: application/json' \
  -H 'Content-Encoding: gzip' \
  --data-binary @-

After a successful request, send a COMPLETE event with the same runId and job identity to verify a complete run lifecycle.

Client Configuration¶

OpenLineage Python Client¶

The OpenLineage Python Client uses the api_key setting to generate a Bearer header. The following configuration also enables gzip and retries for transient errors:

export OPENLINEAGE__TRANSPORT__TYPE=http
export OPENLINEAGE__TRANSPORT__URL='<OPENLINEAGE_BASE_URL>'
export OPENLINEAGE__TRANSPORT__ENDPOINT='/v1/write/openlineage'
export OPENLINEAGE__TRANSPORT__AUTH__TYPE=api_key
export OPENLINEAGE__TRANSPORT__AUTH__APIKEY='<WORKSPACE_TOKEN>'
export OPENLINEAGE__TRANSPORT__COMPRESSION=gzip
export OPENLINEAGE__TRANSPORT__RETRY='{"total":5,"connect":5,"read":5,"backoff_factor":0.3,"status_forcelist":[429,500,502,503,504],"allowed_methods":["POST"]}'

Configuration support varies between client versions. Check the Python Client configuration reference before upgrading.

Apache Airflow¶

Install an OpenLineage Provider version compatible with your Airflow release, then configure the HTTP transport:

pip install apache-airflow-providers-openlineage

export AIRFLOW__OPENLINEAGE__NAMESPACE='production-airflow'
export AIRFLOW__OPENLINEAGE__TRANSPORT='{"type":"http","url":"<OPENLINEAGE_BASE_URL>","endpoint":"/v1/write/openlineage","auth":{"type":"api_key","apiKey":"<WORKSPACE_TOKEN>"},"compression":"gzip"}'

The Provider has specific Airflow version requirements. See the Airflow OpenLineage Provider documentation before installing or upgrading it.

Apache Spark¶

Load an OpenLineage Spark integration package compatible with your Spark release, then set these properties:

spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener
spark.openlineage.transport.type=http
spark.openlineage.transport.url=<OPENLINEAGE_BASE_URL>
spark.openlineage.transport.endpoint=/v1/write/openlineage
spark.openlineage.transport.auth.type=api_key
spark.openlineage.transport.auth.apiKey=<WORKSPACE_TOKEN>
spark.openlineage.transport.compression=gzip
spark.openlineage.namespace=production-spark

For package loading and version compatibility, see the OpenLineage Spark installation guide.

dbt¶

Install openlineage-dbt, reuse the Python Client transport settings, and run dbt-ol instead of dbt:

pip install openlineage-dbt

export OPENLINEAGE__TRANSPORT__TYPE=http
export OPENLINEAGE__TRANSPORT__URL='<OPENLINEAGE_BASE_URL>'
export OPENLINEAGE__TRANSPORT__ENDPOINT='/v1/write/openlineage'
export OPENLINEAGE__TRANSPORT__AUTH__TYPE=api_key
export OPENLINEAGE__TRANSPORT__AUTH__APIKEY='<WORKSPACE_TOKEN>'
export OPENLINEAGE__TRANSPORT__COMPRESSION=gzip
export OPENLINEAGE_NAMESPACE='production-dbt'

dbt-ol run

Do not set only OPENLINEAGE_URL: clients normally default to /api/v1/lineage, while this integration requires an explicit /v1/write/openlineage endpoint. See the OpenLineage dbt integration guide for execution modes and supported adapters.

Responses and Retries¶

Senders must tolerate duplicate delivery and keep a stable runId for the same run. Do not retry deterministic 4xx errors, because invalid events can otherwise remain in the sending queue indefinitely.

Verification and Troubleshooting¶

After configuration, verify the integration in this order:

1. Send a minimal START event with curl and confirm a 2xx response.
2. Send a terminal event with the same run and job identity.
3. In Guance, search for the record by job namespace, job name, or runId.
4. Check producer logs for repeated retries, authentication errors, or serialization errors.

Common issues:

Security Recommendations¶

• Use HTTPS in production and keep certificate verification enabled.
• Prefer Bearer or X-Token; avoid putting tokens in URLs.
• Inject tokens through a Secret, secret manager, or runtime environment. Do not put them in images, configuration templates, shell history, or logs.
• Rotate tokens regularly and revoke an exposed token immediately.
• Facets can contain SQL, schemas, or business metadata. Review sensitive content and collect only what is necessary.

References¶

• OpenLineage core specification
• OpenLineage event examples
• OpenLineage Python Client configuration
• Apache Airflow OpenLineage Provider
• OpenLineage Spark integration
• OpenLineage dbt integration