Skip to main content

Marmot now supports Terraform

· 13 min read · Bruno Schaatsbergen

The Marmot mascot next to the Terraform logo, over the Plugins page

Marmot finds your tables, topics and services by itself. What it can't decide for you is which sources to watch, how often, with which credentials, and who gets to see the result. Those are decisions, and decisions clicked into a UI drift, get forgotten and never go through review.

Marmot ships a Terraform provider so those decisions live in a repository instead. Pipelines, secret stores, service accounts and access grants are resources, declared next to the BigQuery dataset, the IAM role or the Key Vault they refer to, and applied in the same run. It is on the Terraform Registry and the OpenTofu Registry, and it works from Pulumi too.

What belongs in Terraform​

The rule of thumb from the provider's getting started guide: a pipeline discovers, Terraform decides.

Terraform declaresA pipeline discovers
Pipelines: which plugin, which source, what scheduleAssets: tables, topics, buckets, services
Secret stores and the secrets pipelines readSchemas and columns
Service accounts for agents, CI and ingestionLineage a plugin can see for itself
Access grants: who reaches whatTechnical descriptions
Data products, glossary terms, teams and ownership

You can declare assets and lineage by hand with marmot_asset and marmot_lineage, and that is right for a handful of things you own. For a warehouse with ten thousand tables, point a pipeline at it.

The walkthrough below sets up the first three rows on Marmot Cloud: a keyless pipeline, a password read from your own vault, and an access grant. Pick your cloud in each step. The examples are the ones from the Google Cloud, AWS and Azure guides, which also show every step in the console and the CLI.


Step 1: Point the provider at your instance​

terraform {
required_version = ">= 1.11"

required_providers {
marmot = {
source = "marmotdata/marmot"
}
}
}

provider "marmot" {
host = "https://<instance>.marmotdata.cloud"
}

The block has no credential in it, and usually that's right. The provider takes an API key from api_key or MARMOT_API_KEY, a bearer token from token or MARMOT_TOKEN, and with neither it uses the session marmot login wrote. So on your machine you sign in once, and in CI you set one environment variable.

marmot login https://<instance>.marmotdata.cloud
terraform plan

The CLI keeps a short-lived token, so there is no key on disk.


Step 2: Declare a keyless pipeline​

On Marmot Cloud a pipeline signs in to your cloud as itself, with no key, as keyless auth explains. In Terraform that is the trust on your cloud's side and the pipeline on Marmot's, in one configuration. The marmot_pipeline resource exposes issuer and subject, so the grant can reference the pipeline and a rename updates both sides in one apply.

A Workload Identity pool that trusts your instance, a BigQuery Metadata Viewer grant to the pipeline's principal, and the pipeline:

provider "google" {
project = "analytics-prod"
}

resource "google_iam_workload_identity_pool" "marmot" {
workload_identity_pool_id = "marmot"
}

locals {
pool = google_iam_workload_identity_pool.marmot
principal = "principal://iam.googleapis.com/${local.pool.name}/subject"
}

resource "google_iam_workload_identity_pool_provider" "marmot" {
workload_identity_pool_id = local.pool.workload_identity_pool_id
workload_identity_pool_provider_id = "marmot"
attribute_mapping = { "google.subject" = "assertion.sub" }

oidc {
issuer_uri = "https://<instance>.marmotdata.cloud"
}
}

resource "google_project_iam_member" "marmot_reads_bigquery" {
project = "analytics-prod"
role = "roles/bigquery.metadataViewer"
member = "${local.principal}/${marmot_pipeline.analytics.subject}"
}

resource "marmot_pipeline" "analytics" {
name = "analytics"
plugin_id = "bigquery"
cron_expression = "0 */6 * * *"

config = jsonencode({
project_id = "analytics-prod"
workload_identity_provider = (
google_iam_workload_identity_pool_provider.marmot.name
)
})
}

The config is whatever the plugin accepts, written in HCL and passed through jsonencode. The server validates it against the plugin at apply time, so an invalid config fails the apply instead of the first run. The plugin directory lists each plugin's settings.


Step 3: Read a password from your vault​

A Postgres on Cloud SQL, RDS or Azure Database can't federate, so its pipeline needs a password. Keep it in your secret manager and declare a secret store: the store is a connection to one secret manager with an identity of its own, secretStore:<name>, and a secret is the address of one value in it. Marmot stores the address, never the value, and reads it right before each run.

The store signs in through the pool from step 2. Grant it Secret Manager Secret Accessor on the one secret, not the project. The subject comes from the store resource, so you don't copy it by hand:

resource "marmot_secret_store_google" "prod" {
name = "gcp-prod"
workload_identity_provider = (
google_iam_workload_identity_pool_provider.marmot.name
)
}

resource "google_secret_manager_secret" "orders_db_password" {
secret_id = "orders-db-password"

replication {
auto {}
}
}

resource "google_secret_manager_secret_iam_member" "marmot_reads_orders" {
secret_id = google_secret_manager_secret.orders_db_password.id
role = "roles/secretmanager.secretAccessor"
member = "${local.principal}/${marmot_secret_store_google.prod.subject}"
}

resource "marmot_secret_store_google_secret" "orders_db_password" {
store = marmot_secret_store_google.prod.id
project = google_secret_manager_secret.orders_db_password.project
secret_id = google_secret_manager_secret.orders_db_password.secret_id
}

resource "marmot_pipeline" "orders" {
name = "orders"
plugin_id = "postgresql"
cron_expression = "0 * * * *"

config = jsonencode({
host = "orders-db.example.com"
user = "marmot_reader"
})

secrets = {
password = marmot_secret_store_google_secret.orders_db_password.id
}
}

The secrets map names a config key and the secret that fills it, and that key is left out of config. The password is not in the configuration, the plan or the state. Rotate it in your secret manager and the next run uses the new value. Terraform doesn't see a change, because it only has the reference.


Step 4: Grant access​

Agents, CI runners and ingestion jobs all talk to the catalog, and none of them should see more than they need. Each one gets a service account, and a new service account can sign in and do nothing else. You add grants from there.

Grants come in three resources per scope, and the difference is how much of the policy Terraform owns. _iam_member adds one member to one role and leaves everything else alone. _iam_binding owns one role and removes members of it that aren't in the configuration. _iam_policy owns the whole policy. Start with _iam_member, as the provider's access control guide recommends. You can grant on the organization, a data product, an asset, a glossary term or a secret store, and grants inherit downwards: a grant on a data product covers the assets in it, now and later.

resource "marmot_service_account" "copilot" {
name = "catalog-copilot"
description = "Answers questions in #data-help. Owned by the platform team."
}

resource "marmot_service_account" "ingest" {
name = "orders-ingest"
description = "Attaches secrets to the orders pipelines. Owned by the platform team."
}

resource "marmot_data_product" "revenue" {
name = "Revenue"
description = "Everything the finance team reconciles against."
}

resource "marmot_organization_iam_member" "everyone_reads" {
role = "catalog.viewer"
member = "allAuthenticated"
}

resource "marmot_data_product_iam_member" "copilot_reads_revenue" {
data_product_id = marmot_data_product.revenue.id
role = "dataProduct.viewer"
member = "serviceAccount:${marmot_service_account.copilot.id}"
}

resource "marmot_secret_store_iam_member" "ingest_uses_prod" {
secret_store_id = marmot_secret_store_google.prod.id
role = "secretStore.user"
member = "serviceAccount:${marmot_service_account.ingest.id}"
}

Everyone signed in can read the catalog. The copilot can also read one data product. The ingest account has no role at the root and can reach one secret store, where secretStore.user lets it attach secrets to pipelines without ever reading a value. Removing that one resource takes the access away. The store is the Google one from step 3; on AWS or Azure, reference marmot_secret_store_aws.prod or marmot_secret_store_azure.prod instead.

There are no deny rules, so keep the catalog-wide grant narrow and lower the base role if the catalog holds something not everyone should read. Policy writes carry an etag: if someone changes a policy in the UI between your plan and your apply, the apply is rejected instead of overwriting their change.


Import what you already have​

Pipelines, stores, service accounts and grants you created in the UI can be brought under Terraform with terraform import. Most resources import by their ID, and grants import by the path in their id:

terraform import marmot_pipeline.analytics <pipeline-id>

terraform import marmot_asset_iam_binding.orders_editors \
"asset/<asset-id>/roles/editor"

Almost every resource can be imported, and its page on the Registry gives the format. Secret stores and access grants are Marmot Cloud and Marmot Enterprise features. Open source Marmot has no secret store or access policy API, so those resources say so at the top of their page and fail on apply there.


OpenTofu, Pulumi and the operator​

The provider is on the OpenTofu Registry with the same source address, so tofu init finds it unchanged. Pulumi uses it through the Terraform provider bridge, with pulumi package add terraform-provider marmotdata/marmot, in the language your Pulumi program already uses; Pulumi shows Go and TypeScript. If you'd rather declare pipelines as Kubernetes manifests and let GitOps apply them, the Kubernetes operator runs each pipeline as its own Job or CronJob.


How the provider is built​

The provider is written in Go on the Terraform Plugin Framework, on top of the Go SDK we also publish for your own tooling. The framework is what makes ephemeral resources possible. Releases are built for Linux, macOS, Windows and FreeBSD. The documentation is generated from the schema and CI fails if it drifts from the code, and the acceptance tests run against two Terraform versions in CI.

Some of us worked on Terraform at HashiCorp, and we learned a lot there about what makes a provider pleasant to use and what makes one frustrating. Credentials can be ephemeral or write-only, so a key never has to land in state. A policy that changed underneath you is rejected instead of overwritten. Every grant says what it removes on apply, and almost every resource says how to import it. When we add something to the Marmot API, the Terraform resource is part of the same work, not a follow-up ticket. If something doesn't work the way you expect, open an issue on the provider repository. We read all of them.

Provider documentation

Every resource, data source and ephemeral resource, with import syntax and examples, plus guides for access control, keyless pipelines, secret stores and CI.

Open the docs(opens in a new tab)

Join the Community

Get help, share feedback and connect with other Marmot users on Discord.

Join Discord(opens in a new tab)