Populating your catalog

There are two ways to fill the catalog. A pipeline points a plugin at a source and discovers what is there on a schedule. Declaring assets as code describes them yourself, next to the infrastructure that creates them. Most catalogs use both: pipelines for the bulk of the catalog, code for the services and APIs no plugin can see.

Start with a pipeline

Point a plugin at a database, warehouse or broker and let it discover your assets.

Pipelines

Discover with plugins

Pipelines

The server runs a plugin on a schedule. Declare it in Terraform or create it in the UI.

UI

Create a pipeline from a form on the Runs page

CLI

Run plugins from any machine or CI job and push the results

Kubernetes operator

Run each pipeline as its own Job or CronJob in your cluster

Declare assets as code

Terraform

Assets, lineage, glossary terms and data products as Terraform resources

Pulumi

The same resources in the language you already use

REST API

Create and update assets from your own tooling

Browse every plugin

Configuration options, required permissions and examples for every source Marmot can catalog.

Plugin registry