Skip to main content
Version: Preview

Send OpenLineage events to Marmot

Point a tool that emits OpenLineage events at Marmot, and its jobs, datasets and lineage appear in the catalog as it runs. For what Marmot does with the events, see OpenLineage.

Before you start​

  • You have a Marmot API key. Create one under Account → API keys with Create API key, or use a service account's key.
  • The machine the producer runs on can reach Marmot over HTTPS.
  • The key is in the producer's environment as MARMOT_API_KEY:
export MARMOT_API_KEY=<api-key>

Point the producer at Marmot​

Give the producer your Marmot URL and the API key. The OpenLineage clients send events to /api/v1/lineage on the URL you give them, with the key as a bearer token:

Set these in the environment of every Airflow scheduler and worker, with the apache-airflow-providers-openlineage package installed:

export OPENLINEAGE_URL=https://marmot.example.com
export OPENLINEAGE_API_KEY="${MARMOT_API_KEY}"
export AIRFLOW__OPENLINEAGE__NAMESPACE=airflow-prod

Each producer's own documentation covers its other options; the OpenLineage documentation links to them all.

Optional: accept events without a key​

When every client on the network is trusted, you can turn authentication off for POST /api/v1/lineage. Add this to the Marmot server config:

openlineage:
auth:
enabled: false

Or set MARMOT_OPENLINEAGE_AUTH_ENABLED=false. The batch endpoint, POST /api/v1/lineage/batch, still needs a key.

Anyone who can reach the endpoint can then write lineage

Only turn authentication off on a network where every client is trusted.

Check that it worked​

  1. Run a job in the producer.
  2. In Marmot, search Discover for the last part of the job's name, such as load for orders_etl.load, and open the job.
  3. Select the Lineage tab. The job's inputs feed into it and its outputs come out of it.
  4. Select the Run history tab. The run shows as Succeeded, Failed or Running; select a run to see its full error.

Troubleshooting​

What you seeWhat to do
The producer logs 401 UnauthorizedCheck that MARMOT_API_KEY holds a key that hasn't expired or been revoked.
The producer logs 403 with Permission deniedThe key's owner needs a role that can manage assets.
The producer logs 400 with Invalid OpenLineage event: run.runId is required, or another required fieldThe producer sent an event Marmot can't file. Upgrade the producer's OpenLineage package.
The producer logs 413 with Request body is over the 10 MiB limitTurn on the client's gzip compression, or turn off large facets such as Spark's logical plan.
A table appears twice, once from the producer and once from its pluginThe producer's namespace isn't an address Marmot recognises, or the two are in different zones. Compare the namespace with the OpenLineage reference and point the producer at the right zone.
A DuckDB statement failed but its run shows Succeededduck_lineage 0.2 reports failed statements as COMPLETE. Check DuckDB's own output for the error.

Next steps​

  • OpenLineage explains how Marmot names datasets, combines runs and traces columns.
  • OpenLineage reference lists every namespace, facet and response.
  • Zones keep two systems of the same kind apart.