Skip to main content
Version: 0.12

OpenLineage

Marmot accepts OpenLineage run events, so a tool that emits them, such as Airflow, dbt or Spark, adds its jobs, datasets and lineage to the catalog as it runs. Support is experimental and hasn't been tested with every producer; report anything that doesn't map correctly on GitHub.

For your own integrations, send lineage with the lineage API instead: it takes edges, down to the column, without the OpenLineage event format.

What gets cataloged​

Each event adds or updates the job and the datasets it read and wrote, links them with lineage and records the run with its status and timing. A dataset Marmot hasn't seen before is added as a stub, so lineage is complete without filling the catalog with undocumented assets.

Events map to these asset types:

Asset typeFrom
DAGAirflow workflows
TaskAirflow tasks
Modeldbt models
Projectdbt projects
TableDatabase tables
FileData files
TopicKafka topics

Authentication​

Producers send events to POST /api/v1/lineage with an API key, either as X-API-Key: <api-key> or as Authorization: Bearer <api-key>. Create a key under Account → API keys with Create API key, or use a service account's key.

To accept events without a key, turn authentication off for this endpoint:

openlineage:
  auth:
    enabled: false

Or set MARMOT_OPENLINEAGE_AUTH_ENABLED=false.

Anyone who can reach the endpoint can then write lineage

Only turn authentication off on a network where every client is trusted.

Configure a producer​

Airflow​

In airflow.cfg:

[openlineage]
transport = http
url = https://marmot.example.com/api/v1/lineage
api_key = <api-key>

dbt​

In profiles.yml:

kelp:
  outputs:
    prod:
      # your connection details
  vars:
    openlineage:
      url: https://marmot.example.com/api/v1/lineage
      api_key: <api-key>

Spark​

export OPENLINEAGE_URL=https://marmot.example.com/api/v1/lineage
export OPENLINEAGE_API_KEY=<api-key>

The OpenLineage documentation covers every producer and its options.

Last updated on