Skip to main content
Version: Preview

OpenLineage reference

Everything Marmot accepts from OpenLineage producers, following OpenLineage 1.55.0 (core schema 2-0-2). For how it fits together, see OpenLineage.

Endpoints​

EndpointTakesAuthentication
POST /api/v1/lineageOne event as JSON, optionally sent with Content-Encoding: gzipAuthorization: Bearer <api-key> or X-API-Key: <api-key>; can be turned off with openlineage.auth.enabled: false
POST /api/v1/lineage/batchA JSON array of events, processed in orderAlways an API key

Event types​

Event typeRecognised byWhat Marmot files
RunEventA schemaURL ending in RunEvent, or a run fieldThe job, its datasets, their lineage and the run
JobEventA schemaURL ending in JobEvent, or a job field and no runThe job, its datasets and their lineage, with no run
DatasetEventA schemaURL ending in DatasetEvent, or a dataset field and no jobThe dataset

Event fields​

FieldRequired onWhen it's missing or unusual
eventTimeEvery eventA time without an offset is read as UTC
run.runIdRunEventAny string is accepted, not only a UUID
job.namespace, job.nameRunEvent, JobEvent
dataset.namespace, dataset.nameDatasetEvent
namespace, name of each input and outputEvery event that lists them
eventTypeNoneA missing or unknown type is read as OTHER
producer, schemaURLNoneThe event type is read from the fields present

Responses​

StatusWhen
200The event was filed. The batch endpoint answers with a summary of what failed
400The body isn't JSON or the event is missing a required field; the message names it. Don't retry
401, 403The API key is missing, invalid or its owner can't manage assets
413The body is over 10 MiB after decompression
415Content-Encoding is something other than gzip
500Marmot couldn't file the event. Retry

Namespaces​

NamespaceNameBecomes
postgres://<host>:<port><database>.<schema>.<table>PostgreSQL table
mysql://<host>:<port>, mariadb://<database>.<table>MySQL or MariaDB table
mssql://<host>:<port>, sqlserver://<database>.<schema>.<table>SQL Server table
oracle://, db2://, teradata://The path the system usesOracle, Db2 or Teradata table
snowflake://<account><database>.<schema>.<table>Snowflake table
redshift://<cluster>.<region>:<port><database>.<schema>.<table>Redshift table
bigquery<project>.<dataset>.<table>BigQuery table
awsathena://athena.<region>.amazonaws.com<catalog>.<database>.<table>Glue table, without the default AwsDataCatalog
arn:aws:glue:<region>:<account>table/<database>/<table>Glue table
hive://, trino://, presto://, clickhouse://, cassandra://, unitycatalog://The path the system usesA table of that system
kafka://<host>:<port><topic>Kafka topic
pubsubtopic:<project>:<topic> or subscription:<project>:<id>Pub/Sub topic or subscription
s3://<bucket>, gs://<bucket><key>File for a data file, Bucket for a prefix
abfss://<container>@<account>.dfs.core.windows.net, wasbs://<path>File or container in Azure Blob Storage
hdfs://<host>:<port>, dbfs://<workspace><path>HDFS or DBFS file or dataset
file, file://<host><path>File, or a dataset for a directory
duckdb://<path><database>.<schema>.<table>DuckDB table, as dbt-duckdb sends
A namespace that isn't an addressAnyThe address in the dataSource or catalog facet, then the 0.13 naming

Facets​

FacetOnWhat Marmot does with it
schemaDatasetSets the asset's columns, in ordinal_position order, with the producer's types and descriptions
columnLineageDatasetColumn mappings on a direct edge from each input to the output
documentationDataset, jobSets the description
ownershipDataset, jobowner and owners in metadata
tagsDataset, jobAsset tags as key:value, leaving out tags on one column
symlinksDatasetEach name leads to one asset
dataSource, catalogDatasetMetadata, and the dataset's address when the namespace isn't one
datasetTypeDatasetSets the asset type, such as View for VIEW
storage, versionDatasetstorage_layer, file_format and version in metadata
lifecycleStateChangeDatasetlifecycle_state in metadata; a finished RENAME moves the asset and a finished DROP removes a stub
dataQualityMetricsDataset, inputrow_count, size_bytes and file_count in metadata
inputStatistics, outputStatisticsInput, outputinput_ and output_ row count, size and file count in metadata
sqlJobSets the job's query; dialect as sql_dialect
sourceCodeJobSets the job's query and its language
sourceCodeLocationJobsource_code_url, repository, path, branch, tag and version in metadata
jobTypeJobSets the asset type and provider; processing_type and integration in metadata
parent, parentRunRunThe job that recorded the parent run contains this job
errorMessage, spark.exceptionRunThe run's error in Run history
Any other facetAnyKept in the run history