Skip to main content

Marmot now supports keyless auth

· 10 min read · Bruno Schaatsbergen

The Marmot mascot next to a crossed-out key, over the Plugins page

Connecting something to BigQuery, Glue and Blob Storage unfortunately often happens through long-lived credentials. A service account key, an access key, a client secret. Each one is a secret that has to be created, stored and rotated, and that lives long enough to leak. Nobody wants to hold them.

Marmot Cloud and Marmot Enterprise implement OIDC to support keyless auth. When a pipeline launches, your instance signs a short-lived token for it, and your cloud trusts that token the way it trusts one from GitHub Actions. The pipeline becomes a principal on Google Cloud, assumes an IAM role on AWS, or signs in as an app registration on Azure. You assign the access by name, and no key is ever created.

How keyless auth works​

Your instance is an OpenID Connect issuer, like GitHub Actions or a Kubernetes cluster. It publishes its signing key at a well-known address, as keyless pipelines explains in full, and each time a pipeline launches it signs a token that names the pipeline and expires in five minutes. Your cloud checks the signature and gives the pipeline the access you assigned to it. Marmot stores no credential.

PlatformA pipeline becomes
Google CloudA principal: principal://iam.googleapis.com/<pool>/subject/pipeline:analytics
AWSA role whose trust policy matches <instance>.marmotdata.cloud:sub to pipeline:lake
AzureA federated credential with subject pipeline:blobs on an app registration or managed identity
HashiCorp VaultA role with bound_subject set to secretStore:vault-prod, for secret stores

Setting it up is two steps on your cloud and one in Marmot: trust your instance, give the pipeline something to become, create the pipeline. Pick your cloud in each step. The examples catalog BigQuery, the Glue Data Catalog and a Blob Storage account, with pipelines named analytics, lake and blobs. The guides for Google Cloud, AWS and Azure have the same steps in the console and in Terraform too.

Everywhere a step asks for the issuer, it is https://<instance>.marmotdata.cloud with no trailing slash, even if your instance has a custom domain.


Step 1: Trust your instance​

Create a Workload Identity pool with one OIDC provider that names your instance as the issuer, and map google.subject to the token's subject. Once per project:

gcloud iam workload-identity-pools create marmot \
--project=analytics-prod --location=global

gcloud iam workload-identity-pools providers create-oidc marmot \
--project=analytics-prod --location=global \
--workload-identity-pool=marmot \
--issuer-uri="https://<instance>.marmotdata.cloud" \
--attribute-mapping="google.subject=assertion.sub"

The later steps refer to the pool as $POOL. It uses the project number, not the project ID:

POOL=projects/<project-number>/locations/global/workloadIdentityPools/marmot

Step 2: Give the pipeline access​

The pipeline will be called analytics, so in the pool it is the principal pipeline:analytics. Grant it BigQuery Metadata Viewer on the project. It reads dataset and table structure, no rows:

gcloud projects add-iam-policy-binding analytics-prod \
--role=roles/bigquery.metadataViewer \
--member="principal://iam.googleapis.com/$POOL/subject/pipeline:analytics"

There is no service account in this step and no key.


Step 3: Create the pipeline​

In your instance, open Plugins and pick the source under Add a source. The thing that makes the pipeline keyless is the same on every cloud: you name the identity it should become instead of giving it a key. Every other Google Cloud and AWS plugin works the same way, and the plugin directory lists each one's settings. On Azure, Blob Storage is the plugin that federates today.

  1. Select Google BigQuery, enter analytics as the Pipeline name and select Continue.
  2. Set Project ID to analytics-prod and Workload Identity Provider to $POOL/providers/marmot, leave the key fields empty and select Check and continue.
  3. Turn on Run on a schedule, set it to 0 */6 * * * and select Create pipeline.
The BigQuery pipeline's connection settings with Project ID analytics-prod and the Workload Identity Provider filled in and no keyThe BigQuery pipeline's connection settings with Project ID analytics-prod and the Workload Identity Provider filled in and no key

Check that it worked​

Open the pipeline under Plugins and select Run now. When the run succeeds, the datasets, tables or containers appear under Discover. From the CLI, marmot runs list --pipelines <name> lists its runs. Each guide has a troubleshooting table for the errors you are most likely to see.

If the first run is denied right after you granted access, wait a few minutes and run it again. IAM bindings on Google Cloud and role assignments on Azure take a moment to propagate. On AWS, every launch is in CloudTrail as an AssumeRoleWithWebIdentity call with the session name marmot and the pipeline's subject, so a denied run is in there too, with the subject the pipeline presented.


Sources that need a password​

A Postgres on RDS or a Kafka cluster with SASL still wants a password. Keep it in Google Secret Manager, AWS Secrets Manager, Azure Key Vault or HashiCorp Vault, and let a secret store read it at launch. The store uses keyless auth too, as secretStore:<name>, trusted the same way as a pipeline, and Marmot keeps the address of the secret, never the value. In the pipeline, choose From secret store for the password field:

A PostgreSQL pipeline's connection settings with the password read from a secret in the gcp-prod storeA PostgreSQL pipeline's connection settings with the password read from a secret in the gcp-prod store

Rotating the password is adding a version in your secret manager. The next run picks it up. Each cloud guide walks through the store for that cloud, and the secret stores page covers Vault, roles and rotation.


Nothing to rotate​

Keyless auth is available on Marmot Cloud and Marmot Enterprise today, for the Google Cloud, AWS and Azure plugins that can federate and for secret stores on Google Secret Manager, AWS Secrets Manager, Azure Key Vault and HashiCorp Vault. Once your cloud trusts your instance, adding a source is a grant on your side and a pipeline on ours. No key is created, so there is nothing to rotate, nothing to leak and nothing to find in a config file a year from now.

Connect your cloud

The full guides for Google Cloud, AWS and Azure, with every step in the console, the CLI and Terraform.

Open the guide

Join the Community

Get help, share feedback and connect with other Marmot users on Discord.

Join Discord(opens in a new tab)