Populating your catalog

There are two ways to fill the catalog. A pipeline points a plugin at a source and discovers what is there on a schedule. Declaring assets as code describes them yourself, next to the infrastructure that creates them. Most catalogs use both: pipelines for the bulk of the catalog, code for the services and APIs no plugin can see.

Start with a pipeline

Point a plugin at a database, warehouse or broker and let it discover your assets.

Pipelines

Discover with plugins

• PipelinesThe server runs a plugin on a schedule. Declare it in Terraform or create it in the UI.
• UICreate a pipeline from a form on the Runs page
• CLIRun plugins from any machine or CI job and push the results
• Kubernetes operatorRun each pipeline as its own Job or CronJob in your cluster

Declare assets as code

• TerraformAssets, lineage, glossary terms and data products as Terraform resources
• PulumiThe same resources in the language you already use
• REST APICreate and update assets from your own tooling

Browse every plugin

Configuration options, required permissions and examples for every source Marmot can catalog.

Plugin registry(opens in a new tab)