Best Data Catalogs for AI Agents in 2026: The AI Context Layer Compared

# Marmot > The open source context layer for agents and humans. Catalog every service, API, queue, topic, database and pipeline, then expose real, governed context to AI agents and your team. This file contains all documentation content in a single document following the llmstxt.org standard. ## Claude Agent SDK The Claude Agent SDK integration ships in the Python and TypeScript SDKs. It has two halves: 1. The **Marmot MCP server** (built into your Marmot instance at `/api/v1/mcp`) gives the agent catalog-aware tools: `discover_data`, `find_ownership`, `lookup_term`. Point the SDK at it via `mcpServers` and it just shows up as a tool source. 2. **`MarmotAgentTracker`** plugs into the SDK's hook system. The first time any hook fires it registers the agent as an asset of type `Agent`, captures every tool output for `mrn://` references, and writes one batched lineage call when the session ends. ## Install ```bash pip install "marmot-sdk[claude-agent]" ``` The `claude-agent` extra adds `claude-agent-sdk`. ```bash pnpm add @marmotdata/sdk @anthropic-ai/claude-agent-sdk ``` `@anthropic-ai/claude-agent-sdk` is loaded only when you import the tracker, so the SDK stays lean for non-agent users. ## Quick start A minimal agent that searches the catalog, registers itself and writes lineage: ```python from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient from marmot import AuthenticatedApiClient, SecurityScheme, mcp_url from marmot.auth import resolve_credential, resolve_host from marmot.integrations import MarmotCatalog from marmot.integrations.claude_agent import MarmotAgentTracker SYSTEM_PROMPT = ( "You answer questions about an organisation's data using the Marmot catalog. " "Always consult the marmot tools; never guess from memory." ) async def main() -> None: client = AuthenticatedApiClient.connect() tracker = MarmotAgentTracker( MarmotCatalog(client), name="catalog-explorer", model="claude-sonnet-4-5", owner="data-eng", ) token = credential.get_token() mcp_headers = ( {"Authorization": f"Bearer {token}"} if credential.scheme is SecurityScheme.bearer else {"X-API-Key": token} ) options = ClaudeAgentOptions( mcp_servers={ "marmot": { "type": "http", "url": mcp_url(client), "headers": mcp_headers, } }, hooks=tracker.hooks(), permission_mode="bypassPermissions", # Without this the agent loads your own settings and CLAUDE.md files, and # answers catalog questions from that context instead of asking Marmot. setting_sources=[], system_prompt=SYSTEM_PROMPT, ) async with ClaudeSDKClient(options=options) as agent: await agent.query("Find a postgres table about orders and summarise it.") async for _ in agent.receive_response(): pass print("agent registered as:", tracker.agent_mrn, file=sys.stderr) asyncio.run(main()) ``` ```ts const { baseUrl, credential } = await resolve({}); const client = new Client({ baseUrl, credential }); const tracker = new MarmotAgentTracker(client, { name: "catalog-explorer", model: "claude-sonnet-4-5", owner: "data-eng", }); for await (const msg of query({ prompt: "Find a postgres table about orders and summarise it.", options: { mcpServers: { marmot: { type: "http", url: `${baseUrl}/api/v1/mcp`, headers: { Authorization: `Bearer ${credential.token}` }, }, }, hooks: tracker.hooks(), permissionMode: "bypassPermissions", }, })) { // stream messages as you wish } console.log("agent registered as:", tracker.agentMrn); ``` After the first run the agent appears in Marmot as `type=Agent`, `service=ClaudeAgent`, `name=catalog-explorer`, with lineage edges from every asset it touched. ## Catalog tools (via MCP) The Marmot MCP server exposes the catalog as tools, namespaced `mcp__marmot__*`: | Tool | Purpose | | ---------------- | ------------------------------------------------------------- | | `discover_data` | Find or browse assets — by name, type, provider, tags, or MRN | | `find_ownership` | Resolve who owns an asset, or what a team/user owns | | `lookup_term` | Search the business glossary for term definitions | Tool responses include MRNs (often inside markdown content blocks). The tracker walks both structured objects and free text for `mrn://` URIs, so lineage is captured automatically. To restrict the agent to a subset: ```python options = ClaudeAgentOptions( mcp_servers={"marmot": {...}}, hooks=tracker.hooks(), setting_sources=[], system_prompt=SYSTEM_PROMPT, allowed_tools=["mcp__marmot__discover_data", "mcp__marmot__find_ownership"], ) ``` ```ts options: { mcpServers: { marmot: { ... } }, hooks: tracker.hooks(), allowedTools: ["mcp__marmot__discover_data", "mcp__marmot__find_ownership"], } ``` ## Custom tools Two ways to attribute lineage from tools you ship as your own MCP server alongside Marmot's. ### MRNs in tool output If your tool returns objects with `mrn` fields — `{ mrn, ... }` or `{ results: [{ mrn, ... }] }` — or text bodies that mention `mrn://...` URIs, the tracker picks them up on every `PostToolUse` hook. This is the same mechanism that captures MRNs from Marmot's MCP responses. ### Manual `record_source` Use this when the upstream is only known at runtime — for example a tool that picks one of several tables: ```python def query_table(table: str, sql: str) -> list[dict]: tracker.record_source(f"mrn://table/postgres/{table}") return run_sql(sql) ``` ```ts function queryTable(table: string, sql: string) { tracker.recordSource(`mrn://table/postgres/${table}`); return runSql(sql); } ``` --- ## Marmot for Agents An agent acting without context is guessing. It doesn't know which table holds customer orders, who owns the payments service, what a column means, or what breaks downstream if it changes something. **Marmot for Agents** ends that context starvation: it plugs your LLM agents into the catalog so they read it for context and write back the lineage they generate. ## What your agents can do ## Two ways to connect ## Supported frameworks --- ## LangChain The LangChain integration ships in the Python and TypeScript SDKs. It has two halves: 1. **`catalog_tools(client)`** returns a list of LangChain tools (`search_catalog`, `get_asset`, `lookup_asset`, `get_upstream_lineage`) bound to your Marmot client. Drop them into any agent. 2. **`MarmotCallbackHandler`** registers the agent on first run as an asset of type `Agent`, captures every tool call and writes one batched lineage edge per upstream when the run ends. ## Install ```bash pip install "marmot-sdk[langchain]" ``` The `langchain` extra adds `langchain-core`. The agent runtime and model providers are up to you. ```bash pnpm add @marmotdata/sdk @langchain/core ``` `@langchain/core` is an optional peer dependency, so the SDK stays lean for non-agent users. ## Quick start A minimal agent that searches the catalog, registers itself and writes lineage: ```python from langchain_core.messages import HumanMessage, SystemMessage from langchain_core.runnables import RunnableConfig, RunnableLambda from langchain_openai import ChatOpenAI from marmot import AuthenticatedApiClient from marmot.auth import resolve_credential, resolve_host from marmot.integrations import MarmotCatalog from marmot.integrations.langchain import MarmotCallbackHandler, catalog_tools AGENT_NAME = "catalog-explorer" MODEL = "gpt-4o-mini" SYSTEM_PROMPT = ( "You answer questions about an organisation's data using the Marmot catalog. " "Always consult the marmot tools; never guess from memory." ) catalog = MarmotCatalog(AuthenticatedApiClient.connect()) tools = catalog_tools(catalog) handler = MarmotCallbackHandler( catalog, name=AGENT_NAME, model=MODEL, owner="data-eng", tools=tools, ) llm = ChatOpenAI(model=MODEL, temperature=0).bind_tools(tools) # The handler tracks a *chain* run: it registers the agent when the root chain # starts and flushes on its end, attributing every nested tool and model call to # it. `config` must be forwarded so those nested runs inherit the root. def pipeline(question: str, config: RunnableConfig) -> str: answer = llm.invoke( [SystemMessage(SYSTEM_PROMPT), HumanMessage(question)], config=config ) return str(answer.content) reply = RunnableLambda(pipeline).invoke( "Find a postgres table about orders and summarise it.", config=RunnableConfig(callbacks=[handler]), ) print(reply) print("agent registered as:", handler.agent_mrn) ``` ```ts const client = await connect(); const tools = catalogTools(client); const handler = new MarmotCallbackHandler(client, { name: "catalog-explorer", model: "gpt-4o-mini", owner: "data-eng", tools, }); const agent = createReactAgent({ llm: new ChatOpenAI({ model: "gpt-4o-mini", temperature: 0 }), tools, }); await agent.invoke( { messages: [{ role: "user", content: "Find a postgres table about orders." }] }, { callbacks: [handler] }, ); console.log("agent registered as:", handler.agentMrn); ``` After the first run the agent appears in Marmot as `type=Agent`, `service=LangChain`, `name=catalog-explorer`, with lineage edges from every asset it touched. ## Catalog tools `catalog_tools(client)` returns four tools wrapped around the SDK: | Tool | Purpose | | --- | --- | | `search_catalog` | Find assets by name, description or metadata | | `get_asset` | Fetch full schema and metadata for one asset ID | | `lookup_asset` | Resolve an asset by `(type, service, name)` | | `get_upstream_lineage` | Trace ancestors up to N hops | Their responses include `mrn` fields, so the callback handler picks them up automatically and records the upstreams. ## Custom tools Three ways to attribute lineage from your own tools. ### `marmot_tool` helper ```python from marmot.integrations.langchain import marmot_tool @marmot_tool(asset_mrn="mrn://table/postgres/orders") def query_orders(sql: str) -> list[dict]: """Run a read-only SQL query against the orders table.""" return run_sql(sql) ``` ```ts const queryOrders = marmotTool({ name: "query_orders", description: "Run a SQL query against the orders table.", assetMrn: "mrn://table/postgres/orders", schema: { type: "object", properties: { sql: { type: "string" } }, required: ["sql"], }, func: async ({ sql }: { sql: string }) => runSql(sql), }); ``` The MRN is stamped into tool metadata. The handler reads it on every call. ### Manual `record_source` Use this when the upstream is only known at runtime, for example a tool that picks one of several tables: ```python def query_table(table: str, sql: str) -> list[dict]: handler.record_source(f"mrn://table/postgres/{table}") return run_sql(sql) ``` ```ts function queryTable(table: string, sql: string) { handler.recordSource(`mrn://table/postgres/${table}`); return runSql(sql); } ``` ### MRNs in tool output If your tool returns objects shaped like `{ mrn, ... }` or `{ results: [{ mrn, ... }] }`, the handler walks the output looking for them. This is how `catalog_tools` produces lineage automatically. --- ## Auth0 OIDC Marmot supports Auth0 as an OIDC provider for Single Sign-On authentication. ## Create an Auth0 Application 1. Log in to your Auth0 Dashboard 2. Navigate to **Applications** → **Applications** 3. Click **Create Application** 4. Choose a name for your application (e.g., `Marmot`) 5. Select **Regular Web Applications** as the application type 6. Click **Create** 7. Navigate to the **Settings** tab 8. Configure the following: - **Allowed Callback URLs**: `https://your-marmot-domain.com/auth/auth0/callback` - **Allowed Logout URLs**: `https://your-marmot-domain.com` - **Allowed Web Origins**: `https://your-marmot-domain.com` 9. Click **Save Changes** After creating the application, note: - **Client ID**: Found on the **Settings** tab - **Client Secret**: Found on the **Settings** tab - **Domain**: Your Auth0 domain (e.g., `https://dev-12345.us.auth0.com`) ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` Then set the following environment variables: ```bash export MARMOT_AUTH_AUTH0_ENABLED=true export MARMOT_AUTH_AUTH0_CLIENT_ID="your-client-id" export MARMOT_AUTH_AUTH0_CLIENT_SECRET="your-client-secret" export MARMOT_AUTH_AUTH0_URL="https://dev-12345.us.auth0.com" ``` Or configure via `config.yaml`: ```yaml auth: auth0: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" url: "https://dev-12345.us.auth0.com" ``` Restart Marmot and the Auth0 login button will appear on the login page. ## Team Synchronisation Marmot can automatically sync users to teams based on Auth0 group memberships. Enable team sync: ```yaml auth: auth0: team_sync: enabled: true strip_prefix: "marmot-" group: claim: "groups" filter: mode: "include" pattern: "^marmot-.*" ``` To include groups in the ID token: 1. In your Auth0 application, navigate to **Actions** → **Flows** 2. Select **Login** 3. Click **Custom** and create a new action 4. Add the following code: ```javascript exports.onExecutePostLogin = async (event, api) => { if (event.authorization) { api.idToken.setCustomClaim('groups', event.user.groups || []); } }; ``` 5. Deploy the action and add it to your Login flow 6. Ensure your user has groups assigned in Auth0 Alternatively, you can add groups via **Auth0 Authorization Extension** or **User Metadata**. ## Custom TLS Configuration If your Auth0 instance uses a self-signed certificate or a certificate signed by an internal CA (e.g. Auth0 Private Cloud), you can configure Marmot to trust it: ```yaml auth: auth0: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" url: "https://auth.internal" tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" ``` Or via environment variables: ```bash export MARMOT_AUTH_AUTH0_TLS_CA_CERT_PATH="/etc/ssl/certs/internal-ca.pem" ``` To skip TLS verification entirely (not recommended for production): ```bash export MARMOT_AUTH_AUTH0_TLS_INSECURE_SKIP_VERIFY=true ``` If your Auth0 instance requires mutual TLS (mTLS), you can provide a client certificate and key: ```yaml auth: auth0: tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" cert_path: "/etc/ssl/certs/client.pem" key_path: "/etc/ssl/private/client-key.pem" ``` | Field | Description | |-------|-------------| | `tls.ca_cert_path` | Path to a PEM-encoded CA certificate to trust | | `tls.cert_path` | Path to a PEM-encoded client certificate for mTLS | | `tls.key_path` | Path to the client certificate's private key | | `tls.insecure_skip_verify` | Skip TLS certificate verification (default: `false`) | --- ## Generic OIDC Marmot supports any OIDC-compliant identity provider for Single Sign-On authentication. Use this provider when your identity provider is not listed as a dedicated integration. You will need a **Client ID**, **Client Secret** and **Issuer URL** from your identity provider. The redirect URI to register is `https://your-marmot-domain.com/auth/generic_oidc/callback`. ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` Then set the following environment variables: ```bash export MARMOT_AUTH_GENERIC_OIDC_ENABLED=true export MARMOT_AUTH_GENERIC_OIDC_CLIENT_ID="your-client-id" export MARMOT_AUTH_GENERIC_OIDC_CLIENT_SECRET="your-client-secret" export MARMOT_AUTH_GENERIC_OIDC_URL="https://idp.example.com/realms/my-org" ``` Or configure via `config.yaml`: ```yaml auth: generic_oidc: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" url: "https://idp.example.com/realms/my-org" ``` The `url` field is the OIDC issuer URL. Marmot appends `/.well-known/openid-configuration` to discover endpoints automatically. Restart Marmot and the SSO login button will appear on the login page. ### Custom Display Name By default the login button reads "Sign in with SSO". You can change this with the `name` field: ```bash export MARMOT_AUTH_GENERIC_OIDC_NAME="Corporate Login" ``` ## Team Synchronisation Marmot can automatically sync users to teams based on group claims from your identity provider. Enable team sync: ```yaml auth: generic_oidc: team_sync: enabled: true strip_prefix: "/" group: claim: "groups" filter: mode: "include" pattern: ".*" ``` Your identity provider must include a `groups` claim (or your chosen claim name) in the ID token or userinfo response. Consult your identity provider's documentation for how to configure group claims. ## Custom TLS Configuration If your identity provider uses a self-signed certificate or a certificate signed by an internal CA, you can configure Marmot to trust it: ```yaml auth: generic_oidc: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" url: "https://idp.internal" tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" ``` Or via environment variables: ```bash export MARMOT_AUTH_GENERIC_OIDC_TLS_CA_CERT_PATH="/etc/ssl/certs/internal-ca.pem" ``` To skip TLS verification entirely (not recommended for production): ```bash export MARMOT_AUTH_GENERIC_OIDC_TLS_INSECURE_SKIP_VERIFY=true ``` If your identity provider requires mutual TLS (mTLS), you can provide a client certificate and key: ```yaml auth: generic_oidc: tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" cert_path: "/etc/ssl/certs/client.pem" key_path: "/etc/ssl/private/client-key.pem" ``` | Field | Description | |-------|-------------| | `tls.ca_cert_path` | Path to a PEM-encoded CA certificate to trust | | `tls.cert_path` | Path to a PEM-encoded client certificate for mTLS | | `tls.key_path` | Path to the client certificate's private key | | `tls.insecure_skip_verify` | Skip TLS certificate verification (default: `false`) | --- ## GitHub OAuth Marmot supports GitHub as an OAuth provider for Single Sign-On authentication. ## Create a GitHub OAuth App 1. Log in to GitHub 2. Navigate to **Settings** → **Developer settings** → **OAuth Apps** - Personal accounts: [https://github.com/settings/developers](https://github.com/settings/developers) - Organisations: `https://github.com/organisations/YOUR-ORG/settings/applications` 3. Click **New OAuth App** 4. Fill in the application details: - **Application name**: `Marmot` - **Homepage URL**: `https://your-marmot-domain.com` - **Authorisation callback URL**: `https://your-marmot-domain.com/auth/github/callback` 5. Click **Register application** ## Generate Client Secret 1. On the application page, click **Generate a new client secret** 2. Copy the client secret immediately Note the **Client ID** and **Client Secret** from the application page. ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` Then set the following environment variables: ```bash export MARMOT_AUTH_GITHUB_ENABLED=true export MARMOT_AUTH_GITHUB_CLIENT_ID="your-client-id" export MARMOT_AUTH_GITHUB_CLIENT_SECRET="your-client-secret" ``` Or configure via `config.yaml`: ```yaml auth: github: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" ``` Restart Marmot and the GitHub login button will appear on the login page. --- ## GitLab OIDC Marmot supports GitLab (both gitlab.com and self-hosted) as an OIDC provider for Single Sign-On authentication. ## Create a GitLab Application 1. Log in to your GitLab instance 2. Navigate to **User Settings** → **Applications** (or for groups: **Settings** → **Applications**) - GitLab.com: [https://gitlab.com/-/profile/applications](https://gitlab.com/-/profile/applications) - Self-hosted: `https://your-gitlab-instance.com/-/profile/applications` 3. Click **Add new application** 4. Fill in the application details: - **Name**: `Marmot` - **Redirect URI**: `https://your-marmot-domain.com/auth/gitlab/callback` - **Confidential**: Check this option - **Scopes**: Select `openid`, `profile`, and `email` 5. Click **Save application** After creating the application, note: - **Application ID**: Your client ID - **Secret**: Your client secret ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` ### For GitLab.com Then set the following environment variables: ```bash export MARMOT_AUTH_GITLAB_ENABLED=true export MARMOT_AUTH_GITLAB_CLIENT_ID="your-application-id" export MARMOT_AUTH_GITLAB_CLIENT_SECRET="your-secret" ``` Or configure via `config.yaml`: ```yaml auth: gitlab: enabled: true client_id: "your-application-id" client_secret: "your-secret" ``` ### For Self-Hosted GitLab If you're using a self-hosted GitLab instance, you need to specify the URL: ```bash export MARMOT_AUTH_GITLAB_ENABLED=true export MARMOT_AUTH_GITLAB_CLIENT_ID="your-application-id" export MARMOT_AUTH_GITLAB_CLIENT_SECRET="your-secret" export MARMOT_AUTH_GITLAB_URL="https://gitlab.your-company.com" ``` Or configure via `config.yaml`: ```yaml auth: gitlab: enabled: true client_id: "your-application-id" client_secret: "your-secret" url: "https://gitlab.your-company.com" ``` Restart Marmot and the GitLab login button will appear on the login page. ## Custom TLS Configuration If your self-hosted GitLab instance uses a self-signed certificate or a certificate signed by an internal CA, you can configure Marmot to trust it: ```yaml auth: gitlab: enabled: true client_id: "your-application-id" client_secret: "your-secret" url: "https://gitlab.internal" tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" ``` Or via environment variables: ```bash export MARMOT_AUTH_GITLAB_TLS_CA_CERT_PATH="/etc/ssl/certs/internal-ca.pem" ``` To skip TLS verification entirely (not recommended for production): ```bash export MARMOT_AUTH_GITLAB_TLS_INSECURE_SKIP_VERIFY=true ``` If your GitLab instance requires mutual TLS (mTLS), you can provide a client certificate and key: ```yaml auth: gitlab: tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" cert_path: "/etc/ssl/certs/client.pem" key_path: "/etc/ssl/private/client-key.pem" ``` | Field | Description | | -------------------------- | ---------------------------------------------------- | | `tls.ca_cert_path` | Path to a PEM-encoded CA certificate to trust | | `tls.cert_path` | Path to a PEM-encoded client certificate for mTLS | | `tls.key_path` | Path to the client certificate's private key | | `tls.insecure_skip_verify` | Skip TLS certificate verification (default: `false`) | --- ## Google OIDC Marmot supports Google as an OIDC provider for Single Sign-On authentication. ## Create a Google Cloud Project 1. Go to the [Google Cloud Console](https://console.cloud.google.com/) 2. Create a new project or select an existing one ## Configure OAuth Consent Screen 1. Navigate to **APIs & Services** → **OAuth consent screen** 2. Select **External** user type (or **Internal** if using Google Workspace) 3. Configure the consent screen: - **App name**: `Marmot` - **User support email**: Your email address - **Developer contact information**: Your email address 4. On the **Scopes** page, add: - `openid` - `.../auth/userinfo.email` - `.../auth/userinfo.profile` 5. Add test users if using External user type ## Create OAuth 2.0 Credentials 1. Navigate to **APIs & Services** → **Credentials** 2. Click **Create Credentials** → **OAuth client ID** 3. Select **Web application** as the application type 4. Configure your client: - **Name**: `Marmot Web Client` - **Authorised JavaScript origins**: `https://your-marmot-domain.com` - **Authorised redirect URIs**: `https://your-marmot-domain.com/auth/google/callback` 5. Click **Create** Note the **Client ID** and **Client Secret** shown in the credentials dialogue. ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` Then set the following environment variables: ```bash export MARMOT_AUTH_GOOGLE_ENABLED=true export MARMOT_AUTH_GOOGLE_CLIENT_ID="your-client-id.apps.googleusercontent.com" export MARMOT_AUTH_GOOGLE_CLIENT_SECRET="your-client-secret" ``` Or configure via `config.yaml`: ```yaml auth: google: enabled: true client_id: "your-client-id.apps.googleusercontent.com" client_secret: "your-client-secret" ``` Restart Marmot and the Google login button will appear on the login page. --- ## Authentication Marmot supports multiple OAuth/OIDC providers for Single Sign-On authentication. You can enable multiple providers simultaneously, and users will see login buttons for all enabled providers. ## Supported Providers ## How It Works ## Prerequisites Before configuring any authentication provider, you must set `server.root_url` to the public URL of your Marmot instance. This is used to generate OAuth callback URLs. ```yaml server: root_url: https://marmot.example.com ``` Or via environment variable: ```bash export MARMOT_SERVER_ROOT_URL=https://marmot.example.com ``` ## Setup Steps Each provider requires: 1. **Set `server.root_url`** - The public URL users access Marmot from 2. **Create OAuth App** - Register an application in the provider's developer console 3. **Configure Marmot** - Add credentials via environment variables or config file 4. **Restart Marmot** - Changes take effect after restart See individual provider guides above for detailed setup instructions. --- ## Keycloak OIDC Marmot supports Keycloak as an OIDC provider for Single Sign-On authentication. ## Create a Keycloak Client 1. Log in to your Keycloak Admin Console 2. Select the realm you want to use (or create a new one) 3. Navigate to **Clients** and click **Create client** 4. Configure the client: - **Client type**: OpenID Connect - **Client ID**: `marmot` (or your preferred ID) 5. On the next step, enable **Client authentication** (this makes it a confidential client) 6. Ensure **Standard flow** is checked 7. Click **Save** 8. Configure the following under **Settings**: - **Valid redirect URIs**: `https://your-marmot-domain.com/auth/keycloak/callback` - **Web origins**: `https://your-marmot-domain.com` 9. Click **Save** After creating the client, note: - **Client ID**: The client ID you chose - **Client Secret**: Found on the **Credentials** tab - **Keycloak URL**: Your Keycloak base URL (e.g., `https://keycloak.example.com`) - **Realm**: The realm name (e.g., `master` or your custom realm) ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` Then set the following environment variables: ```bash export MARMOT_AUTH_KEYCLOAK_ENABLED=true export MARMOT_AUTH_KEYCLOAK_CLIENT_ID="marmot" export MARMOT_AUTH_KEYCLOAK_CLIENT_SECRET="your-client-secret" export MARMOT_AUTH_KEYCLOAK_URL="https://keycloak.example.com" export MARMOT_AUTH_KEYCLOAK_REALM="your-realm" ``` Or configure via `config.yaml`: ```yaml auth: keycloak: enabled: true client_id: "marmot" client_secret: "your-client-secret" url: "https://keycloak.example.com" realm: "your-realm" ``` Marmot constructs the OIDC issuer URL automatically as `{url}/realms/{realm}` and uses OIDC discovery to configure endpoints. Restart Marmot and the Keycloak login button will appear on the login page. ## Team Synchronisation Marmot can automatically sync users to teams based on Keycloak group memberships. Enable team sync: ```yaml auth: keycloak: team_sync: enabled: true strip_prefix: "/" group: claim: "groups" filter: mode: "include" pattern: ".*" ``` :::tip Keycloak prefixes group names with `/` by default (e.g., `/engineering`). Use `strip_prefix: "/"` to remove this prefix so teams are created as `engineering` instead of `/engineering`. ::: To include groups in the ID token: 1. In your Keycloak Admin Console, navigate to **Clients** and select your Marmot client 2. Go to the **Client scopes** tab 3. Click on the `marmot-dedicated` scope (or your client's dedicated scope) 4. Click **Add mapper** → **By configuration** 5. Select **Group Membership** 6. Configure the mapper: - **Name**: `groups` - **Token Claim Name**: `groups` - **Full group path**: Off (recommended to get simple group names) - **Add to ID token**: On - **Add to access token**: On - **Add to userinfo**: On 7. Click **Save** ## Custom TLS Configuration If your Keycloak instance uses a self-signed certificate or a certificate signed by an internal CA, you can configure Marmot to trust it: ```yaml auth: keycloak: enabled: true client_id: "marmot" client_secret: "your-client-secret" url: "https://keycloak.internal" realm: "your-realm" tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" ``` Or via environment variables: ```bash export MARMOT_AUTH_KEYCLOAK_TLS_CA_CERT_PATH="/etc/ssl/certs/internal-ca.pem" ``` To skip TLS verification entirely (not recommended for production): ```bash export MARMOT_AUTH_KEYCLOAK_TLS_INSECURE_SKIP_VERIFY=true ``` If your Keycloak instance requires mutual TLS (mTLS), you can provide a client certificate and key: ```yaml auth: keycloak: tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" cert_path: "/etc/ssl/certs/client.pem" key_path: "/etc/ssl/private/client-key.pem" ``` | Field | Description | |-------|-------------| | `tls.ca_cert_path` | Path to a PEM-encoded CA certificate to trust | | `tls.cert_path` | Path to a PEM-encoded client certificate for mTLS | | `tls.key_path` | Path to the client certificate's private key | | `tls.insecure_skip_verify` | Skip TLS certificate verification (default: `false`) | --- ## Okta OIDC Marmot supports Okta as an OIDC provider for Single Sign-On authentication. ## Create an Okta Application 1. Log in to your Okta Admin Console 2. Navigate to **Applications** → **Applications** 3. Click **Create App Integration** 4. Select **OIDC - OpenID Connect** as the sign-in method 5. Select **Web Application** as the application type 6. Configure your application: - **App integration name**: `Marmot` - **Grant type**: Check **Authorization Code** - **Sign-in redirect URIs**: `https://your-marmot-domain.com/auth/okta/callback` - **Sign-out redirect URIs**: `https://your-marmot-domain.com` 7. Click **Save** After creating the application, note: - **Client ID**: Found on the **General** tab - **Client Secret**: Found on the **General** tab - **Okta Domain**: Your Okta organisation URL (e.g., `https://dev-12345.okta.com`) ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` Then set the following environment variables: ```bash export MARMOT_AUTH_OKTA_ENABLED=true export MARMOT_AUTH_OKTA_CLIENT_ID="your-client-id" export MARMOT_AUTH_OKTA_CLIENT_SECRET="your-client-secret" export MARMOT_AUTH_OKTA_URL="https://dev-12345.okta.com" ``` Or configure via `config.yaml`: ```yaml auth: okta: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" url: "https://dev-12345.okta.com" ``` Restart Marmot and the Okta login button will appear on the login page. ## Team Synchronisation Marmot can automatically sync users to teams based on Okta group memberships. Enable team sync: ```yaml auth: okta: team_sync: enabled: true strip_prefix: "marmot-" group: claim: "groups" filter: mode: "include" pattern: "^marmot-.*" ``` To include groups in the ID token: 1. In your Okta application, go to **Sign On** tab 2. Click **Edit** next to **OpenID Connect ID Token** 3. Under **Groups claim type**, select **Filter** 4. Configure the filter with claim name `groups` and pattern `.*` ## Custom TLS Configuration If your Okta instance uses a self-signed certificate or a certificate signed by an internal CA (e.g. Okta on-prem or via a proxy), you can configure Marmot to trust it: ```yaml auth: okta: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" url: "https://okta.internal" tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" ``` Or via environment variables: ```bash export MARMOT_AUTH_OKTA_TLS_CA_CERT_PATH="/etc/ssl/certs/internal-ca.pem" ``` To skip TLS verification entirely (not recommended for production): ```bash export MARMOT_AUTH_OKTA_TLS_INSECURE_SKIP_VERIFY=true ``` If your Okta instance requires mutual TLS (mTLS), you can provide a client certificate and key: ```yaml auth: okta: tls: ca_cert_path: "/etc/ssl/certs/internal-ca.pem" cert_path: "/etc/ssl/certs/client.pem" key_path: "/etc/ssl/private/client-key.pem" ``` | Field | Description | | -------------------------- | ---------------------------------------------------- | | `tls.ca_cert_path` | Path to a PEM-encoded CA certificate to trust | | `tls.cert_path` | Path to a PEM-encoded client certificate for mTLS | | `tls.key_path` | Path to the client certificate's private key | | `tls.insecure_skip_verify` | Skip TLS certificate verification (default: `false`) | --- ## Slack OIDC Marmot supports Slack as an OIDC provider for Single Sign-On authentication, allowing users to sign in with their Slack workspace credentials. ## Create a Slack App 1. Go to [https://api.slack.com/apps](https://api.slack.com/apps) 2. Click **Create New App** 3. Select **From scratch** 4. Enter the following details: - **App Name**: `Marmot` - **Pick a workspace to develop your app in**: Select your workspace 5. Click **Create App** ## Configure OAuth & Permissions 1. In your app's settings, navigate to **OAuth & Permissions** 2. Under **Redirect URLs**, click **Add New Redirect URL** 3. Add: `https://your-marmot-domain.com/auth/slack/callback` 4. Click **Save URLs** ## Add OAuth Scopes 1. Scroll down to **Scopes** section 2. Under **User Token Scopes**, add the following scopes: - `openid` - `profile` - `email` ## Get Client Credentials 1. Navigate to **Basic Information** in the sidebar 2. Under **App Credentials**, you'll find: - **Client ID**: Your application's client ID - **Client Secret**: Click **Show** to reveal your client secret ## Configure Marmot You must set `MARMOT_SERVER_ROOT_URL` to the public URL of your Marmot instance. This is used to build OAuth callback URLs. ```bash export MARMOT_SERVER_ROOT_URL="https://marmot.example.com" ``` Then set the following environment variables: ```bash export MARMOT_AUTH_SLACK_ENABLED=true export MARMOT_AUTH_SLACK_CLIENT_ID="your-client-id" export MARMOT_AUTH_SLACK_CLIENT_SECRET="your-client-secret" ``` Or configure via `config.yaml`: ```yaml auth: slack: enabled: true client_id: "your-client-id" client_secret: "your-client-secret" ``` Restart Marmot and the Slack login button will appear on the login page. ## Notes - Users must be members of the Slack workspace where the app is installed - Email addresses from Slack will be used to match or create user accounts in Marmot - Make sure users have verified email addresses in their Slack profiles --- ## Anonymous Authentication Marmot allows you to enable anonymous authentication, which provides restricted access to the application without requiring users to log in. When anonymous authentication is enabled, users can still log in with other authentication methods to access their full permissions. Anonymous users will only have access to endpoints that match the permissions of their assigned role. ## Configuration Anonymous authentication can be enabled using either YAML configuration or environment variables. ### YAML Add the following to your `config.yaml` file: ```yaml auth: anonymous: enabled: true ``` ### Environment Variables Set these environment variables: ``` MARMOT_AUTH_ANONYMOUS_ENABLED=true ``` ## Options | Option | Description | Default | | --------- | ------------------------------------------- | ------- | | `enabled` | Whether anonymous authentication is enabled | `false` | | `role` | The role to assign to anonymous users | `user` | --- ## Customisable Banner Marmot allows you to display a customisable banner at the top of the application to communicate important information to users, such as maintenance notices, announcements, or warnings. ## Configuration The banner can be configured using either YAML configuration or environment variables. ### YAML Add the following to your `config.yaml` file: ```yaml ui: banner: enabled: true dismissible: true variant: info message: Welcome to Marmot! id: welcome-banner ``` ### Environment Variables Set these environment variables: ``` MARMOT_UI_BANNER_ENABLED=true MARMOT_UI_BANNER_DISMISSIBLE=true MARMOT_UI_BANNER_VARIANT=info MARMOT_UI_BANNER_MESSAGE=Welcome to Marmot! MARMOT_UI_BANNER_ID=welcome-banner ``` ## Options | Option | Description | Default | Values | | ------------- | -------------------------------------------------------------------- | ------- | --------------------------------------- | | `enabled` | Whether the banner is displayed | `false` | `true`, `false` | | `dismissible` | Whether users can dismiss the banner | `false` | `true`, `false` | | `variant` | Visual style of the banner | `info` | `info`, `warning`, `error`, `success` | | `message` | Message to display in the banner | - | Any string | | `id` | Unique identifier for the banner (used for tracking dismissal state) | - | Any string (e.g., `maintenance-jan-25`) | ## Variants The banner supports four visual variants: - **info**: Blue banner for general information and announcements - **warning**: Orange banner for warnings and important notices - **error**: Red banner for critical alerts and errors - **success**: Green banner for positive messages and confirmations ## Dismissible Banners When `dismissible` is set to `true`, users can close the banner. The dismissal state is stored locally using the banner's `id`. If you update the `id`, the banner will reappear for all users, even if they previously dismissed it. This is useful for new announcements where you want to ensure all users see the updated message. ## Examples ### Maintenance Notice ```yaml ui: banner: enabled: true dismissible: true variant: warning message: Scheduled maintenance on 25th January, 2025 from 02:00-04:00 GMT id: maintenance-jan-25 ``` ### Critical Alert ```yaml ui: banner: enabled: true dismissible: false variant: error message: Production deployment in progress. Data may be temporarily unavailable. id: prod-deployment-jan-25 ``` ### General Announcement ```yaml ui: banner: enabled: true dismissible: true variant: info message: New features released! Check out the updated glossary and metrics pages. id: release-v2-0 ``` --- ## Elasticsearch Marmot can optionally use Elasticsearch to enhance search with deep fuzzy matching across all record fields including metadata, descriptions, documentation and schemas. ## Configuration ### YAML ```yaml search: elasticsearch: enabled: true addresses: - "http://localhost:9200" index: "marmot" ``` ### Environment Variables ``` MARMOT_SEARCH_ELASTICSEARCH_ENABLED=true MARMOT_SEARCH_ELASTICSEARCH_ADDRESSES=http://localhost:9200 MARMOT_SEARCH_ELASTICSEARCH_INDEX=marmot ``` ## Options | Option | Description | Default | Environment Variable | | --------------------------------------- | -------------------------------------------------------------- | --------------- | ---------------------------------------------- | | `search.elasticsearch.enabled` | Enable Elasticsearch for text search | `false` | `MARMOT_SEARCH_ELASTICSEARCH_ENABLED` | | `search.elasticsearch.addresses` | List of Elasticsearch node URLs | - | `MARMOT_SEARCH_ELASTICSEARCH_ADDRESSES` | | `search.elasticsearch.username` | HTTP Basic Auth username | - | `MARMOT_SEARCH_ELASTICSEARCH_USERNAME` | | `search.elasticsearch.password` | HTTP Basic Auth password | - | `MARMOT_SEARCH_ELASTICSEARCH_PASSWORD` | | `search.elasticsearch.index` | Name of the Elasticsearch index | `marmot` | `MARMOT_SEARCH_ELASTICSEARCH_INDEX` | | `search.elasticsearch.bulk_size` | Number of documents per bulk indexing batch | `500` | `MARMOT_SEARCH_ELASTICSEARCH_BULK_SIZE` | | `search.elasticsearch.flush_interval` | Interval between bulk flushes in milliseconds | `1000` | `MARMOT_SEARCH_ELASTICSEARCH_FLUSH_INTERVAL` | | `search.elasticsearch.reindex_on_start` | Run a full reindex from PostgreSQL to Elasticsearch on startup | `false` | `MARMOT_SEARCH_ELASTICSEARCH_REINDEX_ON_START` | | `search.elasticsearch.shards` | Number of primary shards for the index | cluster default | `MARMOT_SEARCH_ELASTICSEARCH_SHARDS` | | `search.elasticsearch.replicas` | Number of replicas for the index | cluster default | `MARMOT_SEARCH_ELASTICSEARCH_REPLICAS` | ## TLS To connect to an Elasticsearch cluster over TLS: ```yaml search: elasticsearch: enabled: true addresses: - "https://es.example.com:9200" tls: ca_cert_path: "/etc/ssl/certs/es-ca.pem" cert_path: "/etc/ssl/certs/es-client.pem" key_path: "/etc/ssl/private/es-client-key.pem" ``` | Option | Description | Default | Environment Variable | | ----------------------------------------------- | ------------------------------------------------------ | ------- | ------------------------------------------------------ | | `search.elasticsearch.tls.ca_cert_path` | Path to CA certificate for verifying the ES server | - | `MARMOT_SEARCH_ELASTICSEARCH_TLS_CA_CERT_PATH` | | `search.elasticsearch.tls.cert_path` | Path to client certificate for mutual TLS | - | `MARMOT_SEARCH_ELASTICSEARCH_TLS_CERT_PATH` | | `search.elasticsearch.tls.key_path` | Path to client private key for mutual TLS | - | `MARMOT_SEARCH_ELASTICSEARCH_TLS_KEY_PATH` | | `search.elasticsearch.tls.insecure_skip_verify` | Skip server certificate verification (not recommended) | `false` | `MARMOT_SEARCH_ELASTICSEARCH_TLS_INSECURE_SKIP_VERIFY` | ## Startup Behaviour At startup, Marmot checks whether Elasticsearch is reachable. If the cluster is not available, Marmot falls back to PostgreSQL-only search and logs an error. It will not retry connecting to Elasticsearch after startup. ## Shards and Replicas By default Marmot defers to the Elasticsearch cluster settings for shard and replica counts. You can override these per-index if needed: ```yaml search: elasticsearch: enabled: true addresses: - "http://localhost:9200" shards: 3 replicas: 1 ``` These values are only applied when Marmot creates the index for the first time. Changing them after the index already exists has no effect. To apply new shard counts to an existing cluster you must delete the index and let Marmot recreate it, or use the Elasticsearch split/shrink APIs directly. ## Indexing Existing Data If you enable Elasticsearch on an instance that already has data, set `reindex_on_start: true` to populate the index from the existing `search_index` table: ```yaml search: elasticsearch: enabled: true addresses: - "http://localhost:9200" reindex_on_start: true ``` You can also manually trigger a reindex from the admin UI under **Admin > System > Start Reindex**. --- ## Configure Marmot is configured using a YAML file or environment variables. All settings have sensible defaults so you only need to specify what you want to change. ## Configuration Topics ## Configuration File By default, Marmot looks for `config.yaml` in the current directory. Use the `--config` flag to specify a different path. ```yaml database: host: localhost port: 5432 user: postgres password: secret name: marmot server: host: 0.0.0.0 port: 8080 logging: level: info format: json ``` ## Environment Variables All configuration options can be set via environment variables using the `MARMOT_` prefix with underscores separating nested keys. For example, `database.host` becomes `MARMOT_DATABASE_HOST`. ## Database Marmot requires PostgreSQL 14 or later. Ensure the database user has privileges to create tables and indexes. | Key | Description | Default | Environment Variable | | ----------------------- | ---------------------------------------- | ----------- | ------------------------------- | | `database.host` | PostgreSQL host | `localhost` | `MARMOT_DATABASE_HOST` | | `database.port` | PostgreSQL port | `5432` | `MARMOT_DATABASE_PORT` | | `database.user` | Database username | `postgres` | `MARMOT_DATABASE_USER` | | `database.password` | Database password | - | `MARMOT_DATABASE_PASSWORD` | | `database.name` | Database name | `marmot` | `MARMOT_DATABASE_NAME` | | `database.sslmode` | SSL mode (disable, require, verify-full) | `disable` | `MARMOT_DATABASE_SSLMODE` | | `database.maxConns` | Maximum open connections | `10` | `MARMOT_DATABASE_MAX_CONNS` | | `database.idleConns` | Minimum idle connections | `5` | `MARMOT_DATABASE_IDLE_CONNS` | | `database.connLifetime` | Connection lifetime in minutes | `30` | `MARMOT_DATABASE_CONN_LIFETIME` | ## Server | Key | Description | Default | Environment Variable | | ------------------------- | ---------------------------------------------- | --------- | -------------------------------- | | `server.host` | Bind address | `0.0.0.0` | `MARMOT_SERVER_HOST` | | `server.port` | Port number | `8080` | `MARMOT_SERVER_PORT` | | `server.root_url` | Public URL of this Marmot instance | - | `MARMOT_SERVER_ROOT_URL` | | `server.tls.cert_path` | Path to server TLS certificate | - | `MARMOT_SERVER_TLS_CERT_PATH` | | `server.tls.key_path` | Path to server TLS private key | - | `MARMOT_SERVER_TLS_KEY_PATH` | | `server.tls.ca_cert_path` | Path to CA cert for client verification (mTLS) | - | `MARMOT_SERVER_TLS_CA_CERT_PATH` | :::info Root URL Required for Authentication `server.root_url` must be set when using OAuth/OIDC authentication or CLI login (`marmot login`). It is the URL that users access Marmot from (e.g. `https://marmot.example.com`). This is used to generate OAuth callback URLs and redirect users after authentication. ```yaml server: root_url: https://marmot.example.com ``` Or via environment variable: ```bash export MARMOT_SERVER_ROOT_URL=https://marmot.example.com ``` ::: ## Logging Marmot uses structured logging. Set the format to `console` for human-readable output during development. | Key | Description | Default | Environment Variable | | ---------------- | ------------------------------------ | ------- | ----------------------- | | `logging.level` | Log level (debug, info, warn, error) | `info` | `MARMOT_LOGGING_LEVEL` | | `logging.format` | Output format (json, console) | `json` | `MARMOT_LOGGING_FORMAT` | ## Search | Key | Description | Default | Environment Variable | | ---------------- | ---------------------------------------- | ------- | ----------------------- | | `search.timeout` | Search query timeout in seconds | `10` | `MARMOT_SEARCH_TIMEOUT` | See [Elasticsearch](/docs/Configure/elasticsearch) for options related to the optional Elasticsearch search backend. ## OpenLineage | Key | Description | Default | Environment Variable | | -------------------------- | --------------------------------------------------- | ------- | --------------------------------- | | `openlineage.auth.enabled` | Require authentication for the OpenLineage endpoint | `true` | `MARMOT_OPENLINEAGE_AUTH_ENABLED` | --- ## Table Preview Table preview is an **experimental** feature that allows users to see a sample of rows directly in the asset detail page. Because it requires a live connection back to the source database at query time. When enabled, table preview: - Registers the `/api/v1/assets/preview/{id}` API endpoint - Links discovered assets to their ingestion schedules (used to resolve connection details) - Shows a **Preview** tab on table and view assets in the UI ## How to enable ### Configuration file ```yaml experimental: table_preview: true ``` ### Environment variable ```bash export MARMOT_EXPERIMENTAL_TABLE_PREVIEW=true ``` ### Helm chart ```yaml config: experimental: table_preview: true ``` --- ## Telemetry Marmot collects **anonymous** telemetry to help us understand how the product is used and where to focus development effort. Telemetry is enabled by default and can be disabled at any time. ## What is collected - A random install ID (UUID, not tied to any user or organization) - Server version and runtime environment (OS, architecture, Go version) - Deployment mode (Kubernetes, Docker, or binary) - Uptime and the time the report was sent - Aggregate counts: total assets, users, lineage edges, and runs per connector type ## What is never collected - Hostnames, IP addresses, or domain names - Asset names, descriptions, or metadata values - User names, emails, or any PII - Database connection strings or credentials - Query content or API request bodies - Any data that could identify your organization ## How to opt out ### Configuration file ```yaml telemetry: enabled: false ``` ### Helm chart ```yaml config: telemetry: enabled: false ``` ### Environment variable ```bash export MARMOT_TELEMETRY_ENABLED=false ``` --- ## TLS ## Configuration To enable TLS, provide a certificate and private key: ### YAML ```yaml server: port: 8443 tls: cert_path: "/etc/ssl/certs/marmot.pem" key_path: "/etc/ssl/private/marmot-key.pem" ``` ### Environment Variables ``` MARMOT_SERVER_TLS_CERT_PATH=/etc/ssl/certs/marmot.pem MARMOT_SERVER_TLS_KEY_PATH=/etc/ssl/private/marmot-key.pem ``` ## Options | Option | Description | Default | Environment Variable | | ------------------------- | ----------------------------------------------------------------------------- | ------- | -------------------------------- | | `server.tls.cert_path` | Path to a PEM-encoded server certificate | - | `MARMOT_SERVER_TLS_CERT_PATH` | | `server.tls.key_path` | Path to the server certificate's private key | - | `MARMOT_SERVER_TLS_KEY_PATH` | | `server.tls.ca_cert_path` | Path to a PEM-encoded CA certificate for verifying client certificates (mTLS) | - | `MARMOT_SERVER_TLS_CA_CERT_PATH` | Both `cert_path` and `key_path` are required when TLS is enabled. Omitting the `tls` section entirely keeps the server on plain HTTP. ## Mutual TLS (mTLS) To require clients to present a valid certificate, add `ca_cert_path` pointing to the CA that signed your client certificates: ```yaml server: port: 8443 tls: cert_path: "/etc/ssl/certs/marmot.pem" key_path: "/etc/ssl/private/marmot-key.pem" ca_cert_path: "/etc/ssl/certs/client-ca.pem" ``` When `ca_cert_path` is set, the server requires and verifies a client certificate on every request. Clients that do not present a certificate signed by the specified CA will be rejected. --- ## CLI / Binary Run Marmot directly on your system using the single binary. ## Installation --- ## Quick Start ```bash marmot generate-encryption-key ``` Save this key securely. You'll need it to start the server. Create `config.yaml` with your database settings: ```yaml database: host: localhost port: 5432 user: postgres password: your-password name: marmot ``` ```bash export MARMOT_SERVER_ENCRYPTION_KEY="your-generated-key" marmot server --config config.yaml ``` Open [http://localhost:8080](http://localhost:8080) in your browser. The default username and password is **admin:admin**. Change this after your first login. --- ## Development Mode For local development, you can skip encryption (credentials stored in plaintext): ```yaml server: allow_unencrypted: true database: host: localhost port: 5432 user: postgres password: password name: marmot ``` ```bash marmot server --config config.yaml ``` Never use `allow_unencrypted: true` in production environments. --- ## Reference For all configuration options, see the [configuration guide](/docs/Configure). ## Next Steps --- ## Docker Compose Deploy Marmot and PostgreSQL together with Docker Compose. ## Quick Start ```bash curl -fsSL get.marmotdata.io | sh ``` Marmot encrypts sensitive credentials stored in your catalog: ```bash marmot generate-encryption-key ``` Save this key securely. You'll need it in the next step. Create a `docker-compose.yaml`: ```yaml services: marmot: image: ghcr.io/marmotdata/marmot:latest ports: - "8080:8080" environment: MARMOT_DATABASE_HOST: postgres MARMOT_DATABASE_PORT: 5432 MARMOT_DATABASE_USER: marmot MARMOT_DATABASE_PASSWORD: ${POSTGRES_PASSWORD} MARMOT_DATABASE_NAME: marmot MARMOT_DATABASE_SSLMODE: disable MARMOT_SERVER_ENCRYPTION_KEY: ${MARMOT_ENCRYPTION_KEY} depends_on: postgres: condition: service_healthy postgres: image: postgres:16-alpine environment: POSTGRES_USER: marmot POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} POSTGRES_DB: marmot volumes: - marmot_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U marmot"] interval: 5s timeout: 5s retries: 5 volumes: marmot_data: ``` Create a `.env` file in the same directory: ```bash POSTGRES_PASSWORD=your-secure-password MARMOT_ENCRYPTION_KEY=your-generated-key ``` ```bash docker compose up -d ``` Open [http://localhost:8080](http://localhost:8080) in your browser. The default username and password is **admin:admin**. Change this after your first login. --- ## Reference For all configuration options, see the [configuration guide](/docs/Configure). ## Next Steps --- ## Docker Deploy Marmot using Docker containers with your own PostgreSQL database. ## Quick Start Install the Marmot CLI and generate a key: ```bash curl -fsSL get.marmotdata.io | sh marmot generate-encryption-key ``` ```bash docker run -d \ --name marmot \ -p 8080:8080 \ -e MARMOT_SERVER_ENCRYPTION_KEY= \ -e MARMOT_DATABASE_HOST= \ -e MARMOT_DATABASE_PORT=5432 \ -e MARMOT_DATABASE_USER= \ -e MARMOT_DATABASE_PASSWORD= \ -e MARMOT_DATABASE_NAME= \ -e MARMOT_DATABASE_SSLMODE=disable \ ghcr.io/marmotdata/marmot:latest ``` Open [http://localhost:8080](http://localhost:8080) in your browser. The default username and password is **admin:admin**. Change this after your first login. --- ## Configuration Pass configuration as environment variables: ```bash docker run -d \ --name marmot \ -p 8080:8080 \ -e MARMOT_SERVER_ENCRYPTION_KEY=your-key \ -e MARMOT_DATABASE_HOST=postgres.example.com \ -e MARMOT_DATABASE_PORT=5432 \ -e MARMOT_DATABASE_USER=marmot \ -e MARMOT_DATABASE_PASSWORD=secret \ -e MARMOT_DATABASE_NAME=marmot \ -e MARMOT_DATABASE_SSLMODE=require \ ghcr.io/marmotdata/marmot:latest ``` Mount a config file for more complex configurations: ```bash docker run -d \ --name marmot \ -p 8080:8080 \ -v /path/to/config.yaml:/app/config.yaml \ ghcr.io/marmotdata/marmot:latest server --config /app/config.yaml ``` Example `config.yaml`: ```yaml server: encryption_key: "your-generated-key" database: host: postgres.example.com port: 5432 user: marmot password: secret name: marmot sslmode: require ``` --- ## Development Mode For local development, you can skip encryption (credentials stored in plaintext): ```bash docker run -d \ --name marmot \ -p 8080:8080 \ -e MARMOT_SERVER_ALLOW_UNENCRYPTED=true \ -e MARMOT_DATABASE_HOST=host.docker.internal \ -e MARMOT_DATABASE_PORT=5432 \ -e MARMOT_DATABASE_USER=postgres \ -e MARMOT_DATABASE_PASSWORD=password \ -e MARMOT_DATABASE_NAME=marmot \ ghcr.io/marmotdata/marmot:latest ``` Never use `MARMOT_SERVER_ALLOW_UNENCRYPTED=true` in production environments. --- ## Reference For all configuration options, see the [configuration guide](/docs/Configure). ## Next Steps --- ## Helm / Kubernetes Deploy Marmot to your Kubernetes cluster using our official Helm chart. ## Quick Start ```bash helm repo add marmotdata https://marmotdata.github.io/charts helm repo update ``` ```bash helm install marmot marmotdata/marmot ``` Port-forward to access the dashboard: ```bash kubectl port-forward svc/marmot 8080:8080 ``` Open [http://localhost:8080](http://localhost:8080) in your browser. The default username and password is **admin:admin**. Change this after your first login. --- ## Database Configuration Marmot requires PostgreSQL. Choose one of the following options: Connect Marmot to your existing PostgreSQL database: ```yaml config: database: host: postgres.example.com port: 5432 user: marmot passwordSecretRef: name: marmot-db-secret key: password name: marmot sslmode: require ``` For development and testing, enable the embedded PostgreSQL: ```bash helm install marmot marmotdata/marmot \ --set postgresql.enabled=true ``` The chart generates a PostgreSQL password Secret named `-postgresql` with a `password` key and reuses it on upgrades. To use an existing Secret: ```yaml postgresql: auth: existingSecret: marmot-postgres-credentials passwordKey: password ``` For production Kubernetes deployments, [CloudNativePG](https://cloudnative-pg.io/) provides a robust PostgreSQL operator with automatic failover, read replicas and connection pooling. Follow the [CloudNativePG installation guide](https://cloudnative-pg.io/documentation/current/installation_upgrade/) to install the operator. The quickest method: ```bash kubectl apply --server-side -f \ https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.25.1.yaml ``` ```yaml # values.yaml cnpg: enabled: true instances: 3 # 1 primary + 2 read replicas password: "your-secure-password" parameters: shared_buffers: "256MB" work_mem: "64MB" effective_cache_size: "1GB" persistence: size: "10Gi" pooler: enabled: true instances: 2 poolMode: "transaction" ``` ```bash helm install marmot marmotdata/marmot -f values.yaml ``` ```bash kubectl get clusters kubectl get pods -l cnpg.io/cluster=marmot-cnpg ``` All pods should show `Running` status with the primary indicated by the `-1` suffix. --- ## Encryption Key Marmot encrypts sensitive pipeline credentials at rest. You must configure an encryption key. The Helm chart can auto-generate an encryption key for you (enabled by default): ```yaml config: server: autoGenerateEncryptionKey: true ``` If the generated secret is deleted, you'll lose access to encrypted credentials. Back it up immediately after installation. Retrieve the auto-generated key: ```bash kubectl get secret -marmot-encryption-key \ -o jsonpath='{.data.encryption-key}' | base64 -d ``` ```bash marmot generate-encryption-key # or openssl rand -base64 32 ``` ```bash kubectl create secret generic marmot-encryption \ --from-literal=encryption-key="your-generated-key" ``` ```yaml config: server: autoGenerateEncryptionKey: false encryptionKeySecretRef: name: marmot-encryption key: encryption-key ``` For local development only, you can disable encryption entirely: ```yaml config: server: autoGenerateEncryptionKey: false allowUnencrypted: true ``` This stores all credentials in plaintext. Never use this in production. --- ## Reference For all available configuration options, view the chart's defaults: ```bash helm show values marmotdata/marmot ``` Or browse the [values.yaml on GitHub](https://github.com/marmotdata/marmot/blob/main/charts/marmot/values.yaml). ## Next Steps --- ## Deploy There are multiple ways to deploy Marmot - choose whichever method works best with your existing infrastructure and workflows. ## Deployment Options ## Next Steps Once deployed, you'll want to populate your catalog with data assets: --- ## Creating a Marmot Plugin This guide walks you through creating a simple HelloWorld plugin for Marmot that demonstrates the core concepts of plugin development. Marmot plugins are standalone binaries built on the [Marmot plugin SDK](https://github.com/marmotdata/plugin-sdk). Marmot launches them on demand via [go-plugin](https://github.com/hashicorp/go-plugin) and talks to them over gRPC: once at startup to read their metadata, then once per run to validate configuration and discover assets. Your plugin lives in its own repository, with its own dependencies and release cycle. [marmot-plugin-gcs](https://github.com/marmotdata/marmot-plugin-gcs) is a complete real-world example. ## 1. Create the Plugin Module Create a new Go module and add the SDK: ```bash mkdir marmot-plugin-helloworld && cd marmot-plugin-helloworld go mod init github.com/you/marmot-plugin-helloworld go get github.com/marmotdata/plugin-sdk ``` ## 2. Implement the Source Interface Create `source.go`: ```go package main "context" "fmt" "time" pluginsdk "github.com/marmotdata/plugin-sdk" "github.com/marmotdata/plugin-sdk/mrn" ) // Config for the HelloWorld plugin type Config struct { pluginsdk.BaseConfig `json:",inline"` // Add a simple config option Greeting string `json:"greeting" description:"Optional custom greeting message" default:"Hello!"` } type Source struct { config *Config } // Validate checks if the configuration is valid func (s *Source) Validate(rawConfig pluginsdk.RawConfig) (pluginsdk.RawConfig, error) { config, err := pluginsdk.UnmarshalConfig[Config](rawConfig) if err != nil { return nil, fmt.Errorf("unmarshaling config: %w", err) } // Fill fields from their default tags when absent from the raw config pluginsdk.ApplyDefaults(config, rawConfig) if err := pluginsdk.ValidateStruct(config); err != nil { return nil, err } s.config = config return rawConfig, nil } // Discover creates our hello and world assets. The SDK runs Validate // first in the same process, so s.config is always set here. func (s *Source) Discover(ctx context.Context, rawConfig pluginsdk.RawConfig) (*pluginsdk.DiscoveryResult, error) { helloAsset := createHelloAsset(s.config) worldAsset := createWorldAsset(s.config) // Create lineage between assets lineageEdge := pluginsdk.LineageEdge{ Source: *helloAsset.MRN, Target: *worldAsset.MRN, Type: "PRODUCES", } return &pluginsdk.DiscoveryResult{ Assets: []pluginsdk.Asset{helloAsset, worldAsset}, Lineage: []pluginsdk.LineageEdge{lineageEdge}, }, nil } func createHelloAsset(config *Config) pluginsdk.Asset { name := "hello" mrnValue := mrn.New("Example", "HelloWorld", name) description := "Hello asset created by HelloWorld plugin" metadata := map[string]interface{}{ "type": "foo", } if config.Greeting != "" { metadata["greeting"] = config.Greeting } return pluginsdk.Asset{ Name: &name, MRN: &mrnValue, Type: "Example", Providers: []string{"HelloWorld"}, Description: &description, Metadata: metadata, Tags: config.Tags, Sources: []pluginsdk.AssetSource{{ Name: "HelloWorld", LastSyncAt: time.Now(), Properties: metadata, Priority: 1, }}, } } func createWorldAsset(config *Config) pluginsdk.Asset { name := "world" mrnValue := mrn.New("Example", "HelloWorld", name) description := "World asset created by HelloWorld plugin" metadata := map[string]interface{}{ "type": "bar", } return pluginsdk.Asset{ Name: &name, MRN: &mrnValue, Type: "Example", Providers: []string{"HelloWorld"}, Description: &description, Metadata: metadata, Tags: config.Tags, Sources: []pluginsdk.AssetSource{{ Name: "HelloWorld", LastSyncAt: time.Now(), Properties: metadata, Priority: 1, }}, } } ``` ## 3. Serve the Plugin Create `main.go`. `Serve` hands your source to go-plugin and blocks until Marmot disconnects: ```go package main pluginsdk "github.com/marmotdata/plugin-sdk" ) func main() { pluginsdk.Serve(&pluginsdk.ServeConfig{ Meta: pluginsdk.Meta{ ID: "helloworld", Name: "HelloWorld", Description: "A simple plugin that creates hello and world assets with lineage", Icon: "wave", Category: "example", ConfigSpec: pluginsdk.GenerateConfigSpec(Config{}), }, Source: &Source{}, }) } ``` The metadata defines how your plugin shows up in Marmot: its ID (the source name used in ingest configs), display name, description, icon, category, and the configuration form rendered in the UI. ## 4. Install the Plugin Build the binary and copy it into the directory Marmot scans for local plugins. The binary name must start with `marmot-plugin-`: ```bash go build -o ~/.marmot/plugins/marmot-plugin-helloworld . ``` Marmot discovers it at startup, both the server and the CLI. Set `MARMOT_PLUGINS_DIR` if your Marmot uses a different plugins directory. ## 5. Test the Plugin Create a test configuration file `hello.yaml`: ```yaml name: "helloworld" runs: - helloworld: greeting: "Hello from my first plugin!" tags: - "example" - "hello" ``` Run the ingestion: ```bash marmot ingest -c hello.yaml --host http://localhost:8080 --api-key your-api-key ``` After running, you should see two new assets in your catalog: 1. An asset named "hello" 2. An asset named "world" 3. A lineage relationship showing "hello" produces "world" ## 6. Publish the Plugin Publish your plugin to any OCI registry with [oras](https://oras.land). Build one binary per platform first (GoReleaser works well), then: ```bash REPO=ghcr.io/you/plugins/helloworld VERSION=0.1.0 for platform in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64; do os=${platform%/*} arch=${platform#*/} gzip -c "dist/helloworld_${os}_${arch}/marmot-plugin-helloworld" > marmot-plugin-helloworld oras push "$REPO:$VERSION-$os-$arch" \ --artifact-type application/vnd.marmot.plugin.v1 \ --artifact-platform "$platform" \ marmot-plugin-helloworld:application/vnd.marmot.plugin.v1+gzip done oras manifest index create "$REPO:$VERSION" \ "$VERSION-linux-amd64" "$VERSION-linux-arm64" "$VERSION-darwin-amd64" "$VERSION-darwin-arm64" ``` That pushes one manifest per platform and an index that ties them together. Drop the platforms you do not build for. ### Install a published plugin On the machine running Marmot, pull the binary into the plugins directory and restart Marmot: ```bash tmp=$(mktemp -d) oras pull ghcr.io/you/plugins/helloworld:0.1.0 --platform linux/amd64 -o "$tmp" gunzip -c "$tmp/marmot-plugin-helloworld" > ~/.marmot/plugins/marmot-plugin-helloworld chmod +x ~/.marmot/plugins/marmot-plugin-helloworld ``` Marmot loads every `marmot-plugin-*` binary in `~/.marmot/plugins` (or `MARMOT_PLUGINS_DIR`) at startup, so the plugin is available after a restart. ## Configuration Spec Generation The `pluginsdk.GenerateConfigSpec()` function automatically generates a UI-ready configuration schema from your Config struct using struct tags: ```go type Config struct { pluginsdk.BaseConfig `json:",inline"` // Text input Greeting string `json:"greeting" description:"Custom greeting message"` // Dropdown/select (using oneof validation) Mode string `json:"mode" description:"Operation mode" validate:"oneof=simple advanced"` // Sensitive field (password input) APIKey string `json:"api_key" description:"API authentication key" sensitive:"true"` // Number input with validation Timeout int `json:"timeout" description:"Request timeout in seconds" validate:"min=1,max=300" default:"30"` // Required field Host string `json:"host" description:"Server hostname" validate:"required"` // Nested object TLS *TLSConfig `json:"tls,omitempty" description:"TLS configuration"` } ``` Supported tags: - `json`: Field name in JSON - `description`: Help text shown in UI - `label`: Display label (defaults to a title-cased field name) - `validate`: Validation rules (required, min, max, oneof, etc.) - `sensitive`: Marks field as password/secret - `default`: Default value The `default` tag pre-fills the UI form, but configs written by hand (a CLI ingest file, for example) skip the form. Call `pluginsdk.ApplyDefaults(config, rawConfig)` in `Validate` so the defaults apply either way: it sets every field with a `default` tag whose key is absent from the raw config. Tag values are parsed as JSON (`default:"true"`, `default:"30"`, `default:"[\"a\",\"b\"]"`); values that are not valid JSON apply verbatim to string fields (`default:"production"`). `BaseConfig` adds the standard `tags`, `external_links`, and `filter` fields every plugin supports. Filtering is applied by Marmot after discovery; your plugin only needs to carry the config. ## Column Schema Table-shaped assets can attach a column list that Marmot renders as a tabular "Formatted" view on the asset's Schema tab, with a "Raw" JSON view alongside it. `pluginsdk.SetColumns` marshals a slice of columns into the JSON shape Marmot expects and stores it under the `columns` key of `Asset.Schema`: ```go cols := []pluginsdk.Column{ {Name: "id", DataType: "INTEGER", Nullable: false, PrimaryKey: true, Description: "Surrogate key"}, {Name: "email", DataType: "VARCHAR", Nullable: true}, } if err := pluginsdk.SetColumns(asset, cols); err != nil { return nil, fmt.Errorf("attaching columns: %w", err) } ``` `Column` is a convenience, not a requirement: `SetColumns` accepts a slice of any type that marshals to the recognized shape. To add source-specific fields, embed `Column` in your own type. Embedding flattens it into the same JSON object, so the view reads the canonical fields and the extras stay in the Raw view: ```go type clickhouseColumn struct { pluginsdk.Column Codec string `json:"codec,omitempty"` } cols := []clickhouseColumn{ {Column: pluginsdk.Column{Name: "id", DataType: "UInt64", PrimaryKey: true}}, {Column: pluginsdk.Column{Name: "created_at", DataType: "DateTime"}, Codec: "ZSTD"}, } if err := pluginsdk.SetColumns(asset, cols); err != nil { return nil, fmt.Errorf("attaching columns: %w", err) } ``` which stores under `Asset.Schema["columns"]`: ```json [ { "column_name": "id", "data_type": "UInt64", "is_nullable": false, "is_primary_key": true }, { "column_name": "created_at", "data_type": "DateTime", "is_nullable": false, "codec": "ZSTD" } ] ``` `SetColumns` replaces the column list on every call, so build the whole list and set it once. To mix different column types in one call, a plain `Column` and an extended one, wrap them in a `[]any`: ```go cols := []any{ pluginsdk.Column{Name: "id", DataType: "UInt64", PrimaryKey: true}, clickhouseColumn{Column: pluginsdk.Column{Name: "created_at", DataType: "DateTime"}, Codec: "ZSTD"}, } if err := pluginsdk.SetColumns(asset, cols); err != nil { return nil, fmt.Errorf("attaching columns: %w", err) } ``` Marmot detects the format when the first element has a string `column_name` and a string `data_type`. Recognized keys: | Key | Type | Notes | | --- | --- | --- | | `column_name` | string | Required. The column name. | | `data_type` | string | Rendered as the type badge. Falls back to `unknown` when absent. | | `is_nullable` | bool or string | Drives the Required/Optional badge. Boolean `false`, or the Trino-style string `"NO"`, marks the column Required. Omit the key to show no badge. | | `is_primary_key` | bool | `true` adds a Primary Key annotation. `primary_key` is also accepted. | | `is_sorting_key` | bool | `true` adds a Sorting Key annotation. | | `description` | string | Column description. `comment` is also accepted. | | `default_expression` | any | Shown as the column default. | Only `column_name` and `data_type` are required. Any other keys you emit are preserved in the Raw view but ignored by the Formatted view. `Column` also carries `is_foreign_key` and `is_pii` (both `bool`, omitted when false). They are recorded ahead of column-level lineage and governance work; the Formatted view does not render them yet, so for now they show only in the Raw view. Pick one spelling per key and stay consistent. The accepted aliases (`primary_key` for `is_primary_key`, `comment` for `description`, the string form of `is_nullable`) exist because the core plugins predate a single convention; the Formatted view reads all of them so every plugin renders correctly. New plugins should prefer the keys in the table above. ## Plugin Interface All plugins implement the `pluginsdk.Source` interface: ```go type Source interface { Validate(config RawConfig) (RawConfig, error) Discover(ctx context.Context, config RawConfig) (*DiscoveryResult, error) } ``` **Validate**: Unmarshals and validates configuration before discovery runs **Discover**: Performs the actual asset discovery and returns assets, lineage, and documentation Marmot spawns a fresh plugin process for every call, so `Validate` and `Discover` never share an instance across calls. The SDK runs `Validate` before `Discover` in each process, which means state your `Validate` sets on the `Source` (the parsed config, computed limits) is always there when `Discover` runs. A `Source` can optionally implement `DataFetcher` to power row previews on asset pages: ```go type DataFetcher interface { FetchSampleData(ctx context.Context, config RawConfig, a *Asset) (columnNames []string, rows [][]interface{}, err error) } ``` It gets the asset and the plugin config, queries the source system, and returns column names and sample rows. `Serve` detects the interface automatically, there is nothing to register. ## Write End-to-End Tests The `plugintest` package tests your built binary over the same wire protocol Marmot uses, spawning a fresh process per call just like the host does. That catches problems unit tests miss, like state that does not survive the process model: ```go package main_test "context" "testing" pluginsdk "github.com/marmotdata/plugin-sdk" "github.com/marmotdata/plugin-sdk/plugintest" ) func TestDiscover(t *testing.T) { bin := plugintest.Build(t, ".") // path to the plugin main package result, err := bin.Discover(context.Background(), pluginsdk.RawConfig{ "greeting": "Hello from a test!", }) if err != nil { t.Fatal(err) } if len(result.Assets) != 2 { t.Fatalf("expected 2 assets, got %d", len(result.Assets)) } } ``` `Binary` also exposes `Meta`, `Validate`, and `FetchSampleData`. Pair it with a containerized instance of your source system and the test exercises the exact path Marmot takes in production. ## How Plugins Are Loaded Marmot looks for `marmot-plugin-*` binaries in two places at startup: - `~/.marmot/plugins` (`MARMOT_PLUGINS_DIR`): plugins you installed by hand, like the one in this guide - `~/.marmot/plugins/cache` (`MARMOT_PLUGIN_CACHE_DIR`): core plugins Marmot downloads from `ghcr.io/marmotdata/plugins` Local plugins load first, so a local binary shadows a downloaded core plugin with the same ID. That makes iterating on a core plugin easy: build it into `~/.marmot/plugins` and Marmot runs your build instead of the released one. --- ## Local Development 1. Start PostgreSQL: ```bash docker run --name postgres \ --network bridge \ -e POSTGRES_PASSWORD=postgres \ -e POSTGRES_DB=marmot \ -p 5432:5432 \ -d postgres:latest ``` 2. Start the frontend development server: ```bash cd web/marmot pnpm install pnpm dev ``` 3. In another terminal, start the backend: ```bash make dev-deps make dev ``` The app will be available at: - Frontend: http://localhost:5173 - Backend API: http://localhost:8080 - API Documentation: http://localhost:8080/swagger/index.html --- ## Claude Code Anthropic's CLI for Claude AI with native MCP support. ## Configuration ### Using an API Key Create or edit `~/.claude.json` (user-level) or `.mcp.json` (project root): ```json { "mcpServers": { "marmot": { "type": "http", "url": "https:///api/v1/mcp", "headers": { "X-API-Key": "" } } } } ``` ### Using a Bearer Token If you authenticate with `marmot login`, you can use the cached token instead of an API key: ```json { "mcpServers": { "marmot": { "type": "http", "url": "https:///api/v1/mcp", "headers": { "Authorization": "Bearer " } } } } ``` Project-scoped servers require approval on first use. --- ## Claude Desktop Anthropic's official desktop application with native MCP support. ## Configuration Edit the configuration file for your platform: - **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` - **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` - **Linux**: `~/.config/Claude/claude_desktop_config.json` Add the Marmot MCP server: ```json { "mcpServers": { "marmot": { "command": "npx", "args": [ "-y", "mcp-remote", "https:///api/v1/mcp", "--header", "X-API-Key:" ] } } } ``` For HTTP connections (development), add `--allow-http` to the args array. --- ## Cline VS Code extension for autonomous AI assistance with native MCP support. ## Configuration 1. Open Cline in VS Code (click the Cline icon in the sidebar) 2. Click the MCP Servers icon in Cline's top navigation 3. Select the "Configure" tab 4. Click "Configure MCP Servers" Add the Marmot server to `cline_mcp_settings.json`: ```json { "mcpServers": { "marmot": { "url": "https:///api/v1/mcp", "headers": { "X-API-Key": "" } } } } ``` --- ## Cursor AI-first code editor with native MCP support. ## Configuration ### Global Configuration Create or edit `~/.cursor/mcp.json`: ```json { "mcpServers": { "marmot": { "url": "https:///api/v1/mcp", "headers": { "X-API-Key": "" } } } } ``` ### Project-Level Configuration Create `.cursor/mcp.json` in your project root: ```json { "mcpServers": { "marmot": { "url": "https:///api/v1/mcp", "headers": { "X-API-Key": "" } } } } ``` You can use environment variables with `${env:VAR_NAME}` syntax for sensitive credentials. --- ## Model Context Protocol (MCP) Marmot includes a built-in **Model Context Protocol (MCP)** server that enables AI assistants like Claude, ChatGPT and other LLM-powered tools to interact with your data catalog using natural language. ## What Can You Do? With MCP, you can ask questions like: - "What tables does the analytics team own?" - "Show me all BigQuery datasets tagged as 'production'" - "Find the upstream dependencies for the user_events table" - "Who owns the payment processing API?" ## Choose Your AI Assistant ## Authentication MCP uses the same authentication as Marmot's REST API. You'll need an API key to connect: 1. Navigate to your user profile in Marmot 2. Go to **Settings** → **API Keys** 3. Generate a new API key 4. Use this key in your MCP client configuration The AI assistant will have the same permissions as your user account, respecting all role-based access controls. ## Available Tools Marmot's MCP server provides these tools to AI assistants: ### discover_data Unified data discovery for finding any asset in the catalog. Supports natural language queries, specific lookups by ID or MRN (qualified identifiers like `postgres://db/schema/table`), filtering by type/provider/tags and metadata-based queries. ### find_ownership Bidirectional ownership queries to answer "Who owns this asset?", "What does this user own?" and "Show me all data owned by the data-eng team". Works for both data assets and glossary terms. ### lookup_term Business glossary lookups for understanding terminology and definitions. Search for glossary terms by name or retrieve specific term definitions. --- ## LibreChat Universal AI chat interface supporting multiple providers with native MCP support. ## Configuration Add the Marmot MCP server to your `librechat.yaml`: ```yaml mcpServers: marmot: type: streamable-http url: https:///api/v1/mcp headers: X-API-Key: timeout: 30000 ``` You can use environment variables with `${VAR_NAME}` syntax and user context with `{{LIBRECHAT_USER_ID}}` placeholders. --- ## Skills Give any AI coding agent knowledge of the Marmot CLI, REST API and MCP server. ## Install ```bash npx skills add marmotdata/marmot ``` Once installed, your agent knows how to search assets, explore lineage, look up glossary terms, check ownership and call the Marmot CLI or REST API on your behalf. For the best experience, pair this with [MCP](/docs/MCP) to give the agent a live connection to your Catalog. --- ## Notifications Marmot keeps you informed about changes across your data catalog. Receive notifications when assets are modified, schemas change, pipelines complete, or someone mentions you in documentation. ## Viewing Notifications Click the bell icon in the header to open the notifications panel. Unread notifications appear with a badge showing the count. You can click on the bell icon in the header to read your notifications, mark them as read, or, delete them. ## Subscriptions Beyond ownership-based notifications, you can subscribe to specific assets to receive notifications regardless of whether you own them. When subscribing, choose which notification types you want for that asset. Ownership notifications are automatic - if your team owns an asset, all members receive notifications. Subscriptions let individual users opt in to assets they don't own but want to watch. ## Preferences You can control which notification types you receive globally. Navigate to your profile and find the **Notification Preferences** section. ## Aggregation Marmot batches rapid changes to prevent notification spam. If multiple updates happen to assets you own within a short window, they are grouped into a single notification rather than one per change. Changes are batched within a 2-minute window, with a maximum 5-minute wait before delivery. This keeps your feed manageable during bulk operations and updates. ## External Notifications Send notifications to Slack, Discord, or any HTTP endpoint via team webhooks. --- ## Webhooks Send your team's asset notifications to external services. Each webhook is scoped to a team and can be configured to forward specific notification types. ## Supported Providers ## Setup Open your team page and find the **Webhooks** section. Click **Add Webhook** and fill in the details: - **Name** — a descriptive label (e.g. "Schema alerts to #data-eng") - **Provider** — choose Slack, Discord, or Generic - **Webhook URL** — the incoming webhook URL from your provider - **Notification Types** — select which types to forward Click **Send Test** to verify the webhook is configured correctly. A sample notification will be delivered to your endpoint. ## Provider Details ### Slack Create an incoming webhook in your Slack workspace: 1. Go to [Slack Apps](https://api.slack.com/apps) and create or select an app 2. Enable **Incoming Webhooks** and add a new webhook to your channel 3. Copy the webhook URL (starts with `https://hooks.slack.com/`) Messages are formatted with rich blocks showing the notification type, affected asset, and a link back to Marmot. ### Discord Create a webhook in your Discord server: 1. Open **Server Settings** > **Integrations** > **Webhooks** 2. Click **New Webhook** and select the target channel 3. Copy the webhook URL (starts with `https://discord.com/api/webhooks/`) Notifications are delivered as embedded messages with colour-coded types. ### Generic HTTP For custom integrations, the generic provider sends a JSON POST to any HTTPS endpoint: ```json { "type": "schema_change", "title": "Schema changed on users_table", "message": "Column 'email' type changed from VARCHAR to TEXT", "asset_mrn": "postgres://prod/public/users_table", "team_id": "d4e5f6...", "timestamp": "2025-01-23T10:30:00Z" } ``` --- ## Airflow Experimental Creates: AssetsLineageRun History The Airflow plugin ingests metadata from Apache Airflow, including DAGs (Directed Acyclic Graphs), tasks, and dataset lineage. It connects to Airflow's REST API to discover your orchestration layer and track data dependencies through Airflow's native Dataset feature. ## Prerequisites - **Airflow 2.0+** for basic DAG and task discovery - **Airflow 2.4+** for Dataset-based lineage tracking - REST API enabled with authentication configured :::tip[Authentication] The plugin supports two authentication methods: - **Basic Auth**: Username and password - **API Token**: For token-based authentication Configure authentication in your Airflow instance via `airflow.cfg`: ```ini [api] auth_backends = airflow.api.auth.backend.basic_auth ``` ::: ## Example Configuration ```yaml host: "http://localhost:8080" username: "admin" password: "${AIRFLOW_PASSWORD}" discover_dags: true discover_tasks: true discover_datasets: true include_run_history: true run_history_days: 7 only_active: true filter: include: - "^analytics_.*" exclude: - ".*_test$" tags: - "airflow" - "orchestration" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | api_token | string | false | API token for authentication (alternative to basic auth) | | discover_dags | bool | false | Discover Airflow DAGs as Pipeline assets | | discover_datasets | bool | false | Discover Airflow Datasets for lineage (requires Airflow 2.4+) | | discover_tasks | bool | false | Discover tasks within DAGs | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | Airflow webserver URL (e.g., http://localhost:8080) | | include_run_history | bool | false | Include DAG run history in metadata | | only_active | bool | false | Only discover active (unpaused) DAGs | | password | string | false | Password for basic authentication | | run_history_days | int | false | Number of days of run history to fetch | | tags | TagsConfig | false | Tags to apply to discovered assets | | username | string | false | Username for basic authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | consumer_count | int | Number of DAGs that consume this dataset | | created_at | string | Dataset creation timestamp | | dag_id | string | Unique DAG identifier | | dag_id | string | Parent DAG ID | | dag_run_id | string | Unique identifier for the DAG run | | description | string | DAG description | | downstream_tasks | []string | List of downstream task IDs | | end_date | string | End time of the run | | execution_date | string | Logical execution date | | file_path | string | Path to DAG definition file | | is_active | bool | Whether DAG is active | | is_paused | bool | Whether DAG is paused | | last_parsed_time | string | Last time the DAG file was parsed | | last_run_date | string | Execution date of the last DAG run | | last_run_id | string | ID of the last DAG run | | last_run_state | string | State of the last DAG run (success, failed, running) | | next_run_date | string | Next scheduled run date | | operator_name | string | Airflow operator class name (e.g., BashOperator, PythonOperator) | | owners | string | DAG owners (comma-separated) | | pool | string | Execution pool for the task | | producer_count | int | Number of tasks that produce this dataset | | retries | int | Number of retries configured for the task | | run_count | int | Number of runs in the lookback period | | run_type | string | Type of run (scheduled, manual, backfill) | | schedule_interval | string | DAG schedule (cron expression or preset) | | start_date | string | Actual start time of the run | | state | string | Run state (queued, running, success, failed) | | success_rate | float64 | Success rate percentage over the lookback period | | task_id | string | Task identifier within the DAG | | trigger_rule | string | Task trigger rule (e.g., all_success, one_success) | | updated_at | string | Dataset last update timestamp | | uri | string | Dataset URI identifier | --- ## AsyncAPI Experimental Creates: AssetsLineage ## Example Configuration ```yaml spec_path: "/app/asyncapi-specs" environment: "production" discover_services: true discover_channels: true discover_messages: true tags: - "asyncapi" - "event-driven" filter: include: - "orders.*" - "users.*" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | discover_channels | bool | false | Create channel/topic assets from channels and bindings | | discover_messages | bool | false | Attach message schemas to channel assets | | discover_services | bool | false | Create service assets from AsyncAPI info | | environment | string | false | Environment name (e.g., production, staging) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | git_source | GitSourceConfig | false | Git repository file source configuration | | s3_source | S3SourceConfig | false | S3 file source configuration | | source_type | string | false | File source backend (auto-detected from path when empty) | | spec_path | string | false | Path to AsyncAPI spec file or directory containing specs (local path, s3://bucket/prefix or git::url) | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | allowed_regions | []string | Allowed persistence regions | | asyncapi_version | string | AsyncAPI specification version | | binding_is | string | AMQP binding type (queue or routingKey) | | channel_address | string | Address/topic of the channel | | channel_count | int | Number of channels | | channel_name | string | Name of the channel in the AsyncAPI spec | | cleanup_policy | []string | Topic cleanup policies | | contact_email | string | Contact email address | | contact_name | string | Contact person name | | contact_url | string | Contact URL | | content_deduplication | bool | Whether content-based deduplication is enabled | | deduplication_scope | string | Scope of deduplication if enabled | | delete_retention_ms | int64 | Time to retain deleted messages | | delivery_delay | int | Delivery delay in seconds | | description | string | Description of the resource | | dlq_name | string | Name of the Dead Letter Queue | | environment | string | Environment the resource belongs to | | exchange_auto_delete | bool | Exchange auto delete flag | | exchange_durable | bool | Exchange durability flag | | exchange_name | string | Exchange name | | exchange_type | string | Exchange type (topic, fanout, direct, etc.) | | exchange_vhost | string | Exchange virtual host | | fifo_queue | bool | Whether this is a FIFO queue | | fifo_throughput_limit | string | FIFO throughput limit type | | license | string | License name | | license_url | string | License URL | | max_message_bytes | int | Maximum message size | | max_receive_count | int | Maximum receives before sending to DLQ | | message_retention_duration | string | Message retention duration | | message_retention_period | int | Message retention period in seconds | | operation_count | int | Number of operations | | ordering_type | string | SNS topic ordering type | | partitions | int | Number of partitions | | protocols | []string | List of protocols used | | queue_auto_delete | bool | Queue auto delete flag | | queue_durable | bool | Queue durability flag | | queue_exclusive | bool | Queue exclusivity flag | | queue_name | string | Name of the SQS queue | | queue_name | string | Queue name | | queue_vhost | string | Queue virtual host | | receive_message_wait_time | int | Long polling wait time in seconds | | replicas | int | Number of replicas | | retention_bytes | int64 | Maximum size of the topic | | retention_ms | int64 | Message retention period in milliseconds | | schema_encoding | string | Schema encoding format | | schema_name | string | Schema name | | servers | []string | List of server names | | service_name | string | Name of the service that owns the resource | | service_version | string | Version of the service | | topic_arn | string | SNS Topic ARN | | topic_name | string | Kafka topic name | | topic_name | string | Google Pub/Sub topic name | | topic_name | string | SNS Topic Name | | visibility_timeout | int | Visibility timeout in seconds | --- ## Azure Blob Storage Experimental Creates: Assets The Azure Blob Storage plugin discovers containers from Azure Storage accounts. It captures container metadata including access levels, lease status, and custom metadata. ## Connection Examples ```yaml connection_string: "${AZURE_STORAGE_CONNECTION_STRING}" include_metadata: true tags: - "azure" - "storage" ``` ```yaml account_name: "mystorageaccount" account_key: "${AZURE_STORAGE_ACCOUNT_KEY}" include_metadata: true include_blob_count: false filter: include: - "^data-.*" exclude: - ".*-temp$" tags: - "azure" ``` ## Required Permissions The following Azure RBAC role is recommended: - **Storage Blob Data Reader** - Read access to containers and blobs Or use a custom role with these permissions: - `Microsoft.Storage/storageAccounts/blobServices/containers/read` - `Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read` ## Example Configuration ```yaml connection_string: "${AZURE_STORAGE_CONNECTION_STRING}" include_metadata: true include_blob_count: false filter: include: - "^data-.*" exclude: - ".*-temp$" tags: - "azure" - "storage" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | account_key | string | false | Azure Storage account key | | account_name | string | false | Azure Storage account name | | connection_string | string | false | Azure Storage connection string | | endpoint | string | false | Custom endpoint URL (for Azurite or other emulators) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_blob_count | bool | false | Count blobs in each container (can be slow for large containers) | | include_metadata | bool | false | Include container metadata | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | blob_count | int64 | Number of blobs in the container | | container_name | string | Name of the container | | etag | string | Entity tag for the container | | has_immutability_policy | bool | Whether container has an immutability policy | | has_legal_hold | bool | Whether container has a legal hold | | last_modified | string | Last modification timestamp | | lease_state | string | Lease state (available/leased/expired/breaking/broken) | | lease_status | string | Lease status (locked/unlocked) | | public_access | string | Public access level (none/blob/container) | --- ## BigQuery Experimental Creates: AssetsLineage The BigQuery plugin discovers datasets, tables, views, and external tables from Google BigQuery projects. It captures schemas, statistics, and lineage relationships. ## Required Permissions Assign `roles/bigquery.metadataViewer` to your service account, or these individual permissions: - `bigquery.datasets.get` - `bigquery.tables.get` - `bigquery.tables.list` ## Example Configuration ```yaml project_id: "company-data-warehouse" credentials_path: "/etc/marmot/bq-service-account.json" tags: - "bigquery" - "data-warehouse" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials_json | string | false | Service account credentials JSON content | | credentials_path | string | false | Path to service account credentials JSON file | | exclude_system_datasets | bool | false | Whether to exclude system datasets (_script, _analytics, etc.) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_datasets | bool | false | Whether to discover datasets | | include_external_tables | bool | false | Whether to discover external tables | | include_table_stats | bool | false | Whether to include table statistics (row count, size) | | include_views | bool | false | Whether to discover views | | max_concurrent_requests | int | false | Maximum number of concurrent API requests | | project_id | string | false | Google Cloud Project ID | | tags | TagsConfig | false | Tags to apply to discovered assets | | use_default_credentials | bool | false | Use default Google Cloud credentials | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | access_entries_count | int | Number of access control entries | | clustering_fields | []string | Clustering fields | | creation_time | string | Dataset creation timestamp | | creation_time | string | Table creation timestamp | | dataset_id | string | Dataset ID | | dataset_id | string | Dataset ID | | default_partition_expiration | string | Default partition expiration duration | | default_table_expiration | string | Default table expiration duration | | description | string | Dataset description | | description | string | Column description | | description | string | Table description | | expiration_time | string | Table expiration timestamp | | external_data_config | map[string]interface{} | External data configuration for external tables | | labels | map[string]string | Dataset labels | | labels | map[string]string | Table labels | | last_modified | string | Last modification timestamp | | last_modified | string | Last modification timestamp | | location | string | Geographic location of the dataset | | name | string | Column name | | nested_fields | []map[string]interface{} | Nested fields for RECORD type columns | | num_bytes | int64 | Size of the table in bytes | | num_rows | uint64 | Number of rows in the table | | partition_expiration | string | Partition expiration duration | | project_id | string | Google Cloud Project ID | | project_id | string | Google Cloud Project ID | | range_partitioning_field | string | Range partitioning field | | source_format | string | Source data format (CSV, JSON, AVRO, etc.) | | source_uris | []string | Source URIs for external data | | table_id | string | Table ID | | table_type | string | Table type (TABLE, VIEW, EXTERNAL) | | time_partitioning_field | string | Time partitioning field | | time_partitioning_type | string | Time partitioning type | | type | string | Column data type | | view_query | string | SQL query for views | --- ## ClickHouse Experimental Creates: Assets The ClickHouse plugin discovers databases, tables, and views from ClickHouse instances. It extracts schema information, column details, and table metrics like row counts and storage sizes. ## Connection Examples ```yaml host: "clickhouse.company.com" port: 9000 user: "default" password: "${CLICKHOUSE_PASSWORD}" database: "default" include_databases: true include_columns: true enable_metrics: true tags: - "clickhouse" - "analytics" ``` ```yaml host: "your-instance.clickhouse.cloud" port: 9440 user: "default" password: "${CLICKHOUSE_PASSWORD}" secure: true include_databases: true include_columns: true enable_metrics: true filter: include: - "^analytics.*" exclude: - ".*_temp$" tags: - "clickhouse" - "cloud" ``` ## Required Permissions The user needs read access to system tables: ```sql GRANT SELECT ON system.databases TO marmot_user; GRANT SELECT ON system.tables TO marmot_user; GRANT SELECT ON system.columns TO marmot_user; ``` For read-only discovery of all databases: ```sql GRANT SHOW DATABASES ON *.* TO marmot_user; GRANT SHOW TABLES ON *.* TO marmot_user; GRANT SHOW COLUMNS ON *.* TO marmot_user; ``` ## Example Configuration ```yaml host: "clickhouse.company.com" port: 9000 user: "default" password: "${CLICKHOUSE_PASSWORD}" database: "default" secure: false include_databases: true include_columns: true enable_metrics: true exclude_system_tables: true filter: include: - "^analytics.*" exclude: - ".*_temp$" tags: - "clickhouse" - "analytics" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | database | string | false | Default database to connect to | | enable_metrics | bool | false | Whether to include table metrics (row counts, sizes) | | exclude_system_tables | bool | false | Whether to exclude system tables | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | ClickHouse server hostname or IP address | | include_columns | bool | false | Whether to include column information in table metadata | | include_databases | bool | false | Whether to discover databases | | password | string | false | Password for authentication | | port | int | false | ClickHouse native protocol port | | secure | bool | false | Use TLS/SSL connection | | tags | TagsConfig | false | Tags to apply to discovered assets | | user | string | false | Username for authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | column_name | string | Column name | | comment | string | Database comment/description | | comment | string | Column comment/description | | comment | string | Table comment/description | | data_type | string | Column data type | | database | string | Database name | | database | string | Parent database name | | default_expression | string | Default value expression | | default_kind | string | Default value kind (DEFAULT, MATERIALIZED, ALIAS) | | engine | string | Database engine type | | engine | string | Table engine (MergeTree, ReplacingMergeTree, etc.) | | is_primary_key | bool | Whether column is part of primary key | | is_sorting_key | bool | Whether column is part of sorting key | | row_count | int64 | Estimated row count | | size_bytes | int64 | Table size in bytes | | table_name | string | Table name | --- ## Confluent Cloud Experimental Creates: Assets The Confluent Cloud plugin discovers Kafka topics from Confluent Cloud clusters. It uses the same discovery engine as the Kafka plugin with defaults tuned for Confluent Cloud. Because it is the same engine, topics are catalogued under the Kafka provider and addressed as `mrn://topic/kafka/`, not under a Confluent provider. Running Kafka, Redpanda and Confluent Cloud against clusters that share a topic name will therefore land them on one asset. ## Connection Confluent Cloud requires SASL/SSL authentication with an API key pair. You can create API keys in the Confluent Cloud Console. ```yaml bootstrap_servers: "pkc-xxxxx.us-west-2.aws.confluent.cloud:9092" client_id: "marmot-discovery" authentication: type: "sasl_ssl" username: "your-api-key" password: "your-api-secret" mechanism: "PLAIN" tls: enabled: true ``` ## Schema Registry If your Confluent Cloud environment has Schema Registry enabled, add the following to pull schema metadata: ```yaml schema_registry: url: "https://psrc-xxxxx.us-west-2.aws.confluent.cloud" enabled: true config: basic.auth.user.info: "sr-key:sr-secret" ``` ## Example Configuration ```yaml bootstrap_servers: "kafka-1.prod.com:9092,kafka-2.prod.com:9092" client_id: "marmot-discovery" authentication: type: "sasl_ssl" username: "your-api-key" password: "your-api-secret" mechanism: "PLAIN" tls: enabled: true tags: - "kafka" - "streaming" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | authentication | AuthConfig | false | Authentication configuration | | bootstrap_servers | string | false | Comma-separated list of bootstrap servers | | client_id | string | false | Client ID for the consumer | | client_timeout_seconds | int | false | Request timeout in seconds | | consumer_config | map[string]string | false | Additional consumer configuration | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_partition_info | bool | false | Whether to include partition information in metadata | | include_topic_config | bool | false | Whether to include topic configuration in metadata | | schema_registry | SchemaRegistryConfig | false | Schema Registry configuration | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls | TLSConfig | false | TLS configuration | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | cleanup_policy | string | Topic cleanup policy | | delete_retention_ms | string | Time to retain deleted segments in milliseconds | | key_schema | string | Key schema definition | | key_schema_id | int | ID of the key schema in Schema Registry | | key_schema_type | string | Type of the key schema (AVRO, JSON, etc.) | | key_schema_version | int | Version of the key schema | | max_message_bytes | string | Maximum message size in bytes | | min_insync_replicas | string | Minimum number of in-sync replicas | | partition_count | int32 | Number of partitions | | replication_factor | int16 | Replication factor | | retention_bytes | string | Maximum size of the topic in bytes | | retention_ms | string | Message retention period in milliseconds | | segment_bytes | string | Segment file size in bytes | | segment_ms | string | Segment file roll time in milliseconds | | topic_name | string | Name of the Kafka topic | | value_schema | string | Value schema definition | | value_schema_id | int | ID of the value schema in Schema Registry | | value_schema_type | string | Type of the value schema (AVRO, JSON, etc.) | | value_schema_version | int | Version of the value schema | --- ## DBT Experimental Creates: AssetsLineage The DBT plugin ingests metadata from dbt (Data Build Tool) projects, including models, sources, seeds, and lineage relationships. It reads dbt's generated artifacts to understand your data transformation layer and how it connects to your warehouse. ## Prerequisites Before Marmot can ingest your dbt project, you need to generate the artifact files in your project's `target/` directory. :::warning[Required] Generate `manifest.json` by running: ```bash dbt compile ``` ::: :::tip[Recommended] Generate `catalog.json` for column types and statistics: ```bash dbt docs generate ``` ::: ## File Sources The `target_path` field accepts local paths, S3 URIs (`s3://bucket/prefix`) or Git URIs (`git::https://...`). For S3 and Git sources, the target directory is downloaded to a temporary location before discovery and cleaned up afterwards. See [File Sources](./Shared%20Configuration/File%20Sources.md) for the full list of supported backends, authentication options and configuration examples. ## Example Configuration ```yaml target_path: "/path/to/dbt/project/target" project_name: "analytics" environment: "production" tags: - "dbt" - "analytics" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | discover_models | bool | false | Discover DBT models | | discover_sources | bool | false | Discover DBT sources | | discover_tests | bool | false | Discover DBT tests | | environment | string | false | Environment name (e.g., production, staging) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | git_source | GitSourceConfig | false | Git repository file source configuration | | include_catalog | bool | false | Include catalog.json for table/column descriptions | | include_manifest | bool | false | Include manifest.json for model definitions | | include_run_results | bool | false | Include run_results.json for test results | | include_sources_json | bool | false | Include sources.json for source definitions | | project_name | string | false | DBT project name | | s3_source | S3SourceConfig | false | S3 file source configuration | | source_type | string | false | File source backend (auto-detected from path when empty) | | tags | TagsConfig | false | Tags to apply to discovered assets | | target_path | string | false | Path to DBT target directory containing manifest.json, catalog.json, etc. (local path, s3://bucket/prefix or git::url) | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | adapter_type | string | Database adapter type (postgres, snowflake, bigquery, etc) | | alias | string | Table alias if different from model name | | catalog_comment | string | Comment from database catalog | | column_comment | string | Column comment from database catalog | | column_description | string | Column description from DBT | | column_name | string | Column name | | column_tags | []string | Tags applied to this column | | config_enabled | bool | Whether model is enabled | | config_full_refresh | bool | Whether to perform full refresh | | config_materialized | string | Materialization strategy from config | | config_on_schema_change | string | Behavior when schema changes (append_new_columns, fail, ignore) | | config_persist_docs | bool | Whether to persist documentation to database | | config_tags | string | Tags from config | | data_type | string | Column data type | | database | string | Source database name | | database | string | Target database name | | database | string | Target database name | | dbt_materialized | string | Materialization type (table, view, incremental, ephemeral) | | dbt_original_path | string | Original path in the DBT project | | dbt_package | string | DBT package name | | dbt_package | string | DBT package name | | dbt_package | string | DBT package name | | dbt_path | string | Path to the model file | | dbt_unique_id | string | DBT's unique identifier for this source | | dbt_unique_id | string | DBT's unique identifier for this node | | dbt_unique_id | string | DBT's unique identifier for this seed | | dbt_version | string | DBT version used to generate this model | | environment | string | Deployment environment (dev, prod, etc) | | environment | string | Deployment environment | | environment | string | Deployment environment | | freshness_checked | bool | Whether freshness checks are configured | | fully_qualified_name | string | Fully qualified name (database.schema.table) | | fully_qualified_name | string | Fully qualified name (database.schema.table) | | fully_qualified_name | string | Fully qualified name (database.schema.table) | | identifier | string | Physical table identifier | | last_run_execution_time | float64 | Execution time of last run in seconds | | last_run_failures | int | Number of failures in last run | | last_run_message | string | Message from last DBT run | | last_run_status | string | Status of the last DBT run (success, error, skipped) | | loaded | bool | Whether source was loaded at time of DBT execution | | model_name | string | DBT model name | | owner | string | Table/view owner from database catalog | | project_name | string | DBT project name | | project_name | string | DBT project name | | project_name | string | DBT project name | | raw_sql | string | Raw SQL before compilation | | schema | string | Source schema name | | schema | string | Target schema name | | schema | string | Target schema name | | seed_path | string | Path to seed CSV file | | source_name | string | DBT source name | | stat_approximate_count | int64 | Approximate row count | | stat_bytes | int64 | Size in bytes | | stat_last_modified | string | Last modification timestamp | | stat_num_rows | int64 | Number of rows (alternative) | | stat_row_count | int64 | Number of rows | | stat_size | float64 | Table size | | table_name | string | Physical table/view name in database | | table_name | string | Source table name | | table_name | string | Seed table name | --- ## Delta Lake Experimental Creates: Assets ## Example Configuration ```yaml table_paths: - "/data/delta/events" - "/data/delta/users" tags: - "delta-lake" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | git_source | GitSourceConfig | false | Git repository file source configuration | | s3_source | S3SourceConfig | false | S3 file source configuration | | source_type | string | false | File source backend (auto-detected from path when empty) | | table_paths | []string | false | Paths to Delta Lake table directories (local paths, s3://bucket/prefix or git::url) | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | created_time | int64 | Table creation timestamp in milliseconds | | current_version | int64 | Current Delta log version | | format | string | Data format (e.g. parquet) | | location | string | Table directory path | | min_reader_version | int | Minimum reader protocol version | | min_writer_version | int | Minimum writer protocol version | | num_files | int | Number of active data files | | partition_columns | string | Comma-separated partition column names | | schema_field_count | int | Number of schema fields | | table_id | string | Delta table unique identifier | | total_size | int64 | Total size of active data files in bytes | --- ## DuckDB Experimental Creates: AssetsLineage The DuckDB plugin discovers schemas, tables, views and foreign key relationships from DuckDB database files. ## File Sources The `path` field accepts local paths, S3 URIs (`s3://bucket/key`) or Git URIs (`git::https://...`). For S3 and Git sources, the file is downloaded to a temporary directory before discovery and cleaned up afterwards. See [File Sources](./Shared%20Configuration/File%20Sources.md) for the full list of supported backends, authentication options and configuration examples. ## Example Configuration ```yaml path: "/data/analytics.duckdb" include_columns: true enable_metrics: true discover_foreign_keys: true exclude_system_schemas: true filter: include: - "^main\\..*" exclude: - ".*_temp$" tags: - "duckdb" - "analytics" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | discover_foreign_keys | bool | false | Whether to discover foreign key relationships | | enable_metrics | bool | false | Whether to include table metrics (row counts and sizes) | | exclude_system_schemas | bool | false | Whether to exclude system schemas (information_schema, pg_catalog) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | git_source | GitSourceConfig | false | Git repository file source configuration | | include_columns | bool | false | Whether to include column information in table metadata | | path | string | false | Path to the DuckDB database file (local path, s3://bucket/key or git::url) | | s3_source | S3SourceConfig | false | S3 file source configuration | | source_type | string | false | File source backend (auto-detected from path when empty) | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | column_default | string | Default value expression | | column_name | string | Column name | | comment | string | Object comment/description | | constraint_name | string | Foreign key constraint name | | data_type | string | Column data type | | is_nullable | bool | Whether null values are allowed | | object_type | string | Object type (BASE TABLE, VIEW) | | path | string | Path to the DuckDB database file | | row_count | int64 | Estimated row count | | schema | string | Schema name | | size | int64 | Estimated size in bytes | | source_column | string | Column in the referencing table | | source_schema | string | Schema of the referencing table | | source_table | string | Name of the referencing table | | table_name | string | Table or view name | | target_column | string | Column in the referenced table | | target_schema | string | Schema of the referenced table | | target_table | string | Name of the referenced table | --- ## DynamoDB Experimental Creates: Assets The DynamoDB plugin discovers and catalogs Amazon DynamoDB tables across your AWS accounts. It captures table metadata including key schema, billing mode, indexes, encryption settings, TTL, point-in-time recovery, streams, and tags. ## Required Permissions ## AWS Configuration See [AWS Configuration](./Shared%20Configuration/AWS%20Configuration.md) for the supported AWS configuration options. ## Example Configuration ```yaml credentials: region: "us-east-1" profile: "production" role: "" tags: - "aws" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials | AWSCredentials | false | AWS credentials configuration | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_tags | []string | false | List of AWS tags to include as metadata. By default, all tags are included. | | tags | TagsConfig | false | Tags to apply to discovered assets | | tags_to_metadata | bool | false | Convert AWS tags to Marmot metadata | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | attribute_definitions | string | Attribute definitions for the table's key schema | | billing_mode | string | Billing mode of the table (PROVISIONED or PAY_PER_REQUEST) | | continuous_backups | string | Continuous backups status (ENABLED or DISABLED) | | creation_date | string | Date and time when the table was created | | deletion_protection | string | Whether deletion protection is enabled | | encryption_status | string | Status of server-side encryption | | encryption_type | string | Type of server-side encryption (AES256 or KMS) | | global_table_replicas | string | Regions where global table replicas exist | | gsi_count | int | Number of global secondary indexes | | item_count | int64 | Number of items in the table | | key_schema | string | Key schema of the table (partition and sort keys) | | lsi_count | int | Number of local secondary indexes | | pitr_status | string | Point-in-time recovery status (ENABLED or DISABLED) | | read_capacity_units | int64 | Provisioned read capacity units | | stream_enabled | string | Whether DynamoDB Streams is enabled | | stream_view_type | string | Stream view type (KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, NEW_AND_OLD_IMAGES) | | table_arn | string | The ARN of the DynamoDB table | | table_class | string | Table class (STANDARD or STANDARD_INFREQUENT_ACCESS) | | table_size_bytes | int64 | Total size of the table in bytes | | table_status | string | Current status of the table (ACTIVE, CREATING, etc.) | | tags | map[string]string | AWS resource tags | | ttl_attribute | string | Attribute name used for Time to Live | | ttl_status | string | Time to Live status (ENABLED or DISABLED) | | write_capacity_units | int64 | Provisioned write capacity units | --- ## EKS # Amazon EKS Experimental Creates: AssetsLineageRun History The EKS plugin discovers namespaces, services, deployments, stateful sets, cron jobs, and pods from Amazon EKS clusters. It is the [Kubernetes plugin](./Kubernetes)'s discovery engine with AWS IAM authentication, so the assets, lineage, and run history it produces are identical. See the Kubernetes plugin for details on what gets discovered and how resources are linked. Authentication uses AWS IAM: on each run the plugin mints a short-lived token from the AWS credentials of wherever Marmot runs. There is no static token to store or rotate. This is the clean way to read an EKS cluster from an EC2 instance or another AWS workload. ## Prerequisites Two grants are needed on the AWS side, plus the read-only Kubernetes RBAC role. First, the IAM identity that Marmot runs as needs an [EKS access entry](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) on the cluster (or a mapping in the older `aws-auth` ConfigMap). Second, that access entry must map to a Kubernetes group bound to a read-only role: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: marmot-discovery rules: - apiGroups: [""] resources: ["namespaces", "services", "pods"] verbs: ["get", "list"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "replicasets"] verbs: ["get", "list"] - apiGroups: ["batch"] resources: ["cronjobs", "jobs"] verbs: ["get", "list"] ``` :::tip[AWS credentials] Credentials resolve from the standard AWS chain: IRSA, EKS Pod Identity, an EC2 instance profile, or static keys. Set `credentials.role` to assume a role, or `credentials.region` to pin the region. When Marmot runs outside AWS, set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in its environment and the chain picks them up. ::: ## Connecting to a cluster The plugin looks up the cluster's endpoint and CA certificate from the EKS API, so you only give it the cluster name and region. This needs the `eks:DescribeCluster` permission. ```yaml eks_cluster_name: "prod" credentials: region: "eu-west-1" ``` ## Example Configuration ```yaml eks_cluster_name: "prod" credentials: region: "eu-west-1" namespaces: - "payments" - "orders" discover_pods: false tags: - "kubernetes" - "${labels.team}" ``` The discovery options (`namespaces`, `discover_*`, `cluster_name`, `tags`, and so on) are the same as the [Kubernetes plugin](./Kubernetes); see there for what each one does. The cluster name is used as the asset name prefix unless you set `cluster_name`. ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | annotations_to_metadata | bool | false | Include resource annotations in asset metadata | | cluster_name | string | false | Cluster name to prefix asset names with | | credentials | AWSCredentials | false | AWS credentials configuration | | discover_cronjobs | bool | false | Discover cron jobs, with their recent job runs as run history | | discover_deployments | bool | false | Discover deployments | | discover_namespaces | bool | false | Discover namespaces | | discover_pods | bool | false | Discover pods. Off by default because pods are short-lived and can flood the catalog | | discover_services | bool | false | Discover services | | discover_statefulsets | bool | false | Discover stateful sets | | eks_cluster_name | string | true | EKS cluster name | | exclude_namespaces | []string | false | Namespaces to skip when discovering all namespaces | | label_selector | string | false | Only discover namespaced resources matching this label selector (e.g. team=data) | | labels_to_metadata | bool | false | Include resource labels in asset metadata | | namespaces | []string | false | Namespaces to discover. Empty or ["*"] means all namespaces | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The metadata fields are the same as the [Kubernetes plugin](./Kubernetes#available-metadata). Every asset also carries `cloud` (set to `EKS`), `aws_region`, and `aws_account_id`, so you can tell where a cluster lives without following lineage. The cluster asset additionally carries `cluster_arn`, its canonical AWS identifier. --- ## Elasticsearch(Plugins) Experimental Creates: AssetsLineage The Elasticsearch plugin discovers indices, data streams and aliases from Elasticsearch clusters. ## Required Permissions The connecting user needs `monitor` cluster privilege and `view_index_metadata` on indices. The built-in `viewer` role is usually sufficient. ## Example Configuration ```yaml addresses: - "https://elasticsearch.company.com:9200" username: "elastic" password: "changeme" tags: - "elasticsearch" - "search" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | addresses | []string | false | List of Elasticsearch node URLs | | api_key | string | false | API key for authentication (mutually exclusive with username/password) | | ca_cert_path | string | false | Path to a custom CA certificate file | | cloud_id | string | false | Elastic Cloud ID for connecting to Elastic Cloud | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_aliases | bool | false | Discover aliases | | include_data_streams | bool | false | Discover data streams | | include_index_stats | bool | false | Collect document count and store size metrics | | include_system_indices | bool | false | Include system indices (prefixed with .) | | password | string | false | Password for basic authentication | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls_skip_verify | bool | false | Skip TLS certificate verification | | username | string | false | Username for basic authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | alias_name | string | Name of the alias | | analyzer | string | Analyzer used for the field | | backing_indices | int | Number of backing indices | | cluster | string | Name of the Elasticsearch cluster | | creation_date | string | Date and time when the index was created | | data_stream_name | string | Name of the data stream | | docs_count | int64 | Number of documents in the index | | field_name | string | Full dotted path of the field | | field_type | string | Elasticsearch field type (keyword, text, long, etc.) | | filter_defined | string | Whether a filter is defined on the alias | | generation | int | Current generation of the data stream | | health | string | Health status of the index (green, yellow, red) | | ilm_policy | string | ILM policy applied to the data stream | | index | string | Whether the field is indexed | | index_name | string | Name of the index | | indices | string | Comma-separated list of indices the alias points to | | is_write_index | string | Whether the alias has a designated write index | | replicas | int | Number of replica shards | | shards | int | Number of primary shards | | status | string | Health status of the data stream | | status | string | Open/close status of the index | | store_size | string | Total store size of the index | | template | string | Index template used by the data stream | | timestamp_field | string | Name of the timestamp field | | uuid | string | UUID of the index | --- ## GKE # Google GKE Experimental Creates: AssetsLineageRun History The GKE plugin discovers namespaces, services, deployments, stateful sets, cron jobs, and pods from Google Kubernetes Engine clusters. It is the [Kubernetes plugin](./Kubernetes)'s discovery engine with Google Cloud authentication, so the assets, lineage, and run history it produces are identical. See the Kubernetes plugin for details on what gets discovered and how resources are linked. Authentication uses Google Cloud IAM: on each run the plugin mints a short-lived OAuth token from the Google credentials of wherever Marmot runs. There is no static token to store or rotate. This is the clean way to read a GKE cluster from a GCE instance, Cloud Run, or another Google Cloud workload. ## Prerequisites The identity Marmot runs as needs read access to the cluster, granted two ways: First, a Google Cloud IAM role that allows connecting to the cluster (for example `roles/container.viewer`), so Google authorizes the token. Second, a read-only Kubernetes RBAC role bound to that identity: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: marmot-discovery rules: - apiGroups: [""] resources: ["namespaces", "services", "pods"] verbs: ["get", "list"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "replicasets"] verbs: ["get", "list"] - apiGroups: ["batch"] resources: ["cronjobs", "jobs"] verbs: ["get", "list"] ``` :::tip[Google credentials] Credentials resolve from Application Default Credentials: Workload Identity, a Cloud Run or GCE service account, or `GOOGLE_APPLICATION_CREDENTIALS`. When Marmot runs outside Google Cloud, set `credentials.credentials_json` (or `credentials.credentials_file`) to a service account key. ::: ## Connecting to a cluster Name the cluster and the plugin resolves its endpoint and CA certificate from the GKE management API. Set `project_id`, `location`, and `cluster`. This needs the `container.clusters.get` permission (included in `roles/container.viewer`). ```yaml project_id: "my-project" location: "us-central1" cluster: "autopilot-cluster-1" ``` ## Example Configuration ```yaml project_id: "my-project" location: "us-central1" cluster: "autopilot-cluster-1" namespaces: - "payments" - "orders" discover_pods: false tags: - "kubernetes" - "${labels.team}" ``` The discovery options (`namespaces`, `discover_*`, `cluster_name`, `tags`, and so on) are the same as the [Kubernetes plugin](./Kubernetes); see there for what each one does. The cluster name is used as the asset name prefix unless you set `cluster_name`. ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | annotations_to_metadata | bool | false | Include resource annotations in asset metadata | | cluster | string | true | GKE cluster name | | cluster_name | string | false | Cluster name to prefix asset names with | | credentials | GCPCredentials | false | GCP credentials configuration | | discover_cronjobs | bool | false | Discover cron jobs, with their recent job runs as run history | | discover_deployments | bool | false | Discover deployments | | discover_namespaces | bool | false | Discover namespaces | | discover_pods | bool | false | Discover pods. Off by default because pods are short-lived and can flood the catalog | | discover_services | bool | false | Discover services | | discover_statefulsets | bool | false | Discover stateful sets | | exclude_namespaces | []string | false | Namespaces to skip when discovering all namespaces | | label_selector | string | false | Only discover namespaced resources matching this label selector (e.g. team=data) | | labels_to_metadata | bool | false | Include resource labels in asset metadata | | location | string | true | Cluster region or zone, for example us-central1 | | namespaces | []string | false | Namespaces to discover. Empty or ["*"] means all namespaces | | project_id | string | true | GCP project ID | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The metadata fields are the same as the [Kubernetes plugin](./Kubernetes#available-metadata). Every asset also carries `cloud` (set to `GKE`), `gcp_project`, and `gcp_location`, so you can tell where a cluster lives without following lineage. --- ## Glue Experimental Creates: Assets The Glue plugin discovers and catalogs AWS Glue resources including jobs, databases, tables and crawlers. It captures metadata such as job configurations, table schemas, crawler schedules and database properties. Iceberg-managed tables are automatically skipped (use the dedicated Iceberg plugin instead). ## Required Permissions ## AWS Configuration See [AWS Configuration](./Shared%20Configuration/AWS%20Configuration.md) for the supported AWS configuration options. ## Example Configuration ```yaml credentials: region: "us-east-1" profile: "production" role: "" tags: - "aws" discover_jobs: true discover_databases: true discover_tables: true discover_crawlers: true ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials | AWSCredentials | false | AWS credentials configuration | | discover_crawlers | bool | false | Whether to discover Glue crawlers | | discover_databases | bool | false | Whether to discover Glue databases | | discover_jobs | bool | false | Whether to discover Glue jobs | | discover_tables | bool | false | Whether to discover Glue tables | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_tags | []string | false | List of AWS tags to include as metadata. By default, all tags are included. | | tags | TagsConfig | false | Tags to apply to discovered assets | | tags_to_metadata | bool | false | Convert AWS tags to Marmot metadata | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | catalog_id | string | ID of the Data Catalog | | classification | string | Classification of the table data (csv, parquet, json, etc.) | | classifiers | string | Custom classifiers used by the crawler | | connections | string | Connections used by the job | | create_time | string | Date and time the database was created | | create_time | string | Date and time the table was created | | created_on | string | Date and time the job was created | | creation_time | string | Date and time the crawler was created | | database_name | string | Target database for the crawler | | database_name | string | Name of the database containing the table | | description | string | Description of the database | | glue_version | string | Glue version used by the job | | input_format | string | Hadoop input format class | | last_crawl_error | string | Error message from the last crawl | | last_crawl_status | string | Status of the last crawl | | last_crawl_time | string | Start time of the last crawl | | last_modified_on | string | Date and time the job was last modified | | last_updated | string | Date and time the crawler was last updated | | location | string | S3 location of the table data | | location_uri | string | Location of the database | | max_capacity | float64 | Maximum number of DPU that can be allocated | | max_retries | int | Maximum number of retries | | number_of_workers | int32 | Number of workers allocated to the job | | output_format | string | Hadoop output format class | | owner | string | Owner of the table | | parameters | string | Database parameters | | partition_keys | string | Partition key columns | | recrawl_behavior | string | Recrawl behavior policy | | retention | int32 | Retention period in days | | role | string | IAM role ARN assigned to the job | | role | string | IAM role ARN assigned to the crawler | | schedule | string | Cron schedule expression | | schema_delete_behavior | string | Behavior when schema objects are deleted | | schema_update_behavior | string | Behavior when schema changes are detected | | script_location | string | S3 location of the job script | | security_configuration | string | Security configuration applied to the job | | serde | string | Serialization/deserialization library | | state | string | Current state of the crawler (READY, RUNNING, STOPPING) | | table_type | string | Type of table (EXTERNAL_TABLE, VIRTUAL_VIEW, etc.) | | targets | string | Summary of crawler targets | | timeout | int32 | Job timeout in minutes | | type | string | Job command type (glueetl, pythonshell, gluestreaming) | | update_time | string | Date and time the table was last updated | | worker_type | string | Worker type (Standard, G.1X, G.2X, etc.) | --- ## Google Cloud Storage Experimental Creates: Assets The Google Cloud Storage plugin discovers buckets from GCP projects. It captures bucket metadata including location, storage class, encryption settings, and lifecycle rules. ## Connection Examples ```yaml project_id: "my-gcp-project" credentials_file: "/path/to/service-account.json" include_metadata: true tags: - "gcs" - "storage" ``` ```yaml project_id: "my-gcp-project" credentials_json: "${GCS_CREDENTIALS_JSON}" include_metadata: true include_object_count: false filter: include: - "^data-.*" exclude: - ".*-temp$" tags: - "gcs" ``` ## Required Permissions The service account needs the following IAM roles: - **Storage Bucket Viewer** (`roles/storage.bucketViewer`) - For discovering and listing buckets Or use a custom role with these permissions: - `storage.buckets.list` - `storage.buckets.get` - `storage.objects.list` (if using object count) ## Example Configuration ```yaml project_id: "my-gcp-project" credentials_file: "/path/to/service-account.json" include_metadata: true include_object_count: false filter: include: - "^data-.*" exclude: - ".*-temp$" tags: - "gcs" - "storage" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials_file | string | false | Path to service account JSON file | | credentials_json | string | false | Service account JSON content | | disable_auth | bool | false | Disable authentication (for local emulators) | | endpoint | string | false | Custom endpoint URL (for fake-gcs-server or other emulators) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_metadata | bool | false | Include bucket metadata like labels | | include_object_count | bool | false | Count objects in each bucket (can be slow for large buckets) | | project_id | string | false | Google Cloud project ID | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | bucket_name | string | Name of the bucket | | created | string | Bucket creation timestamp | | encryption | string | Encryption type (google-managed or customer-managed) | | kms_key | string | Customer-managed encryption key name | | lifecycle_rules_count | int | Number of lifecycle rules configured | | location | string | Geographic location of the bucket | | location_type | string | Location type (region, dual-region, multi-region) | | logging_enabled | bool | Whether access logging is enabled | | object_count | int64 | Number of objects in the bucket | | requester_pays | bool | Whether requester pays for access | | retention_period_seconds | int64 | Retention period in seconds | | storage_class | string | Default storage class (STANDARD, NEARLINE, COLDLINE, ARCHIVE) | | versioning | string | Whether object versioning is enabled | --- ## Google Drive Experimental Creates: AssetsLineage The Google Drive plugin catalogues the documents and sheets a team keeps in Drive, so the spreadsheet a report is actually built from is findable next to the warehouse tables feeding it. ## What it discovers | Google Drive | Marmot asset type | |---|---| | the drive itself | Drive | | folder | Folder | | file | File | | Google Sheet | Spreadsheet | | a sheet within a spreadsheet | Table, with its header row as columns | The drive is the root of the tree: folders nest under it, and each file is linked to the folder holding it, so a drive is navigable from either end: the **Contents** tab on a folder lists what is inside it, and on a file it shows the folder it lives in. These are the same asset types and names the [OpenMetadata](./OpenMetadata) plugin produces for a Google Drive service. An organisation moving off OpenMetadata can import their drive from there and later point this plugin at Drive directly; the second run takes over the assets that are already in the catalog rather than creating a second copy. ## Access The plugin reads Drive with a Google service account. A service account on its own only sees files that have been **shared with its email address**, which is a reasonable way to catalogue a handful of shared folders. To read a whole organisation's Drive, give the service account [domain-wide delegation](https://developers.google.com/workspace/guides/create-credentials#optional_set_up_domain-wide_delegation_for_a_service_account) and set `impersonate_user` to a Workspace user to act as. Enable the **Google Drive API**, and the **Google Sheets API** if you want the columns of each sheet. The scopes needed are: ``` https://www.googleapis.com/auth/drive.metadata.readonly https://www.googleapis.com/auth/spreadsheets.readonly ``` Credentials follow Marmot's shared Google Cloud configuration: a key file, key JSON, or Application Default Credentials when nothing is set. ## Example Configuration ```yaml credentials: credentials_file: "/etc/marmot/gcp-service-account.json" impersonate_user: "data-platform@company.com" tags: - "google-drive" ``` A single shared drive: ```yaml credentials: credentials_file: "/etc/marmot/gcp-service-account.json" drive_id: "0AItAbCdEfGhIjKlMnO" exclude_mime_types: - "application/vnd.google-apps.shortcut" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | concurrency | int | false | Parallel requests for the sheets of a spreadsheet | | credentials | GCPCredentials | false | GCP credentials configuration | | drive_id | string | false | Shared drive to read. Empty means the user's My Drive | | exclude_mime_types | []string | false | MIME types to skip, for example application/vnd.google-apps.shortcut | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | folder_id | string | false | Only discover this folder and everything under it | | header_row | int | false | Row of a sheet that holds the column names | | impersonate_user | string | false | Workspace user to act as. Needed to read a whole organisation's Drive, and requires domain-wide delegation on the service account. Without it, only files shared with the service account are visible | | include_files | bool | false | Discover files, not just folders | | include_spreadsheets | bool | false | Discover Google Sheets, including each of their sheets | | include_trashed | bool | false | Discover files in the trash | | include_worksheets | bool | false | Discover each sheet of a spreadsheet as a table, with its columns. Needs the Sheets read scope | | max_files | int | false | Stop after this many files. 0 means no limit | | page_size | int | false | Files per API request | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | checksum | string | File checksum reported by Drive | | column_count | int | Number of columns in a sheet | | drive_id | string | Google Drive file id | | file_extension | string | File extension | | file_type | string | Broad kind of file, for example Document, Spreadsheet or Image | | file_version | string | Drive version number | | hidden | bool | Whether the sheet is hidden | | mime_type | string | File MIME type | | modified_at | string | When the item last changed in Drive | | owners | []string | People who own the item in Drive | | path | string | Location within the drive | | row_count | int64 | Number of rows in a sheet | | shared | bool | Whether the item is shared | | size | int64 | Size in bytes | | spreadsheet | string | Spreadsheet a sheet belongs to | --- ## Iceberg Experimental Creates: AssetsLineage The Iceberg plugin discovers namespaces, tables and views from Iceberg catalogs. It supports both REST catalogs and AWS Glue Data Catalog as backends. ## AWS Glue Catalog Permissions When using `catalog_type: "glue"`, the following IAM permissions are required: The `s3:GetObject` permission is needed because Glue's `LoadTable` reads Iceberg metadata files from S3. ## AWS Configuration When using `catalog_type: "glue"`, see [AWS Configuration](./Shared%20Configuration/AWS%20Configuration.md) for the supported AWS configuration options. ## Example Configuration ```yaml # REST catalog (default) uri: "http://localhost:8181" warehouse: "my-warehouse" credential: "client-id:client-secret" tags: - "iceberg" # Glue catalog: # catalog_type: "glue" # credentials: # region: "us-east-1" # glue_catalog_id: "123456789012" # optional, defaults to caller's account ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | catalog_type | string | false | Catalog backend type | | credential | string | false | Credential for OAuth2 client credentials authentication | | credentials | AWSCredentials | false | AWS credentials configuration | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | glue_catalog_id | string | false | AWS Glue Data Catalog ID (defaults to caller's account) | | include_namespaces | bool | false | Whether to discover namespaces as assets | | include_tags | []string | false | List of AWS tags to include as metadata. By default, all tags are included. | | include_views | bool | false | Whether to discover views | | prefix | string | false | Optional prefix for the REST catalog | | properties | map[string]string | false | Additional catalog properties | | tags | TagsConfig | false | Tags to apply to discovered assets | | tags_to_metadata | bool | false | Convert AWS tags to Marmot metadata | | token | string | false | Bearer token for authentication | | uri | string | false | REST catalog URI (required for catalog_type=rest) | | warehouse | string | false | Warehouse identifier | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | current_snapshot_id | string | Current snapshot ID | | format_version | int | Iceberg format version (1, 2, or 3) | | format_version | int | View format version | | last_updated_ms | int64 | Last update timestamp in milliseconds | | location | string | Table data location | | location | string | Default location for tables | | location | string | View metadata location | | namespace | string | Namespace path | | partition_spec | string | Partition specification | | schema_field_count | int | Number of schema fields | | schema_field_count | int | Number of schema fields | | snapshot_count | int | Number of snapshots | | sort_order | string | Sort order specification | | sql | string | SQL definition of the view | | sql_dialect | string | SQL dialect of the view definition | | table_uuid | string | Table UUID | | total_data_files | string | Total data file count | | total_file_size | string | Total file size in bytes | | total_records | string | Total record count | | view_uuid | string | View UUID | --- ## Kafka Experimental Creates: Assets Marmot plugin for Apache Kafka. Discovers topics from Kafka clusters, captures topic configurations and partition details and optionally enriches assets with schemas from a Confluent Schema Registry. Marmot plugins are standalone binaries that the Marmot host launches on demand via [go-plugin](https://github.com/hashicorp/go-plugin) and talks to over gRPC. It is built on the [Marmot plugin SDK](https://github.com/marmotdata/plugin-sdk). > Looking for a managed service? Marmot has dedicated plugins for [Confluent Cloud](./Confluent%20Cloud) and [Redpanda](./Redpanda) with pre-configured defaults. ## Connection Examples ### Self-Hosted with SASL ```yaml bootstrap_servers: "kafka-1.prod.com:9092,kafka-2.prod.com:9092" client_id: "marmot-discovery" authentication: type: "sasl_ssl" username: "your-username" password: "your-password" mechanism: "SCRAM-SHA-512" tls: enabled: true ca_cert_path: "/path/to/ca.pem" cert_path: "/path/to/client.pem" key_path: "/path/to/client-key.pem" ``` ### Self-Hosted with mTLS ```yaml bootstrap_servers: "kafka-1.internal:9093" client_id: "marmot-discovery" tls: enabled: true ca_cert_path: "/etc/kafka/ca.pem" cert_path: "/etc/kafka/client.pem" key_path: "/etc/kafka/client-key.pem" ``` ### Local development (no auth) ```yaml bootstrap_servers: "localhost:9092" client_id: "marmot-discovery" tls: enabled: false ``` ## Schema Registry Enable Schema Registry to enrich discovered topics with their value and key schemas: ```yaml schema_registry: enabled: true url: "https://schema-registry.prod.com" config: basic.auth.user.info: "sr-key:sr-secret" ``` Schemas for subjects matching `{topic}-value`, `{topic}-key` or other `{topic}-*` patterns are pulled from the registry and attached to the topic asset. ## Example Configuration ```yaml bootstrap_servers: "kafka-1.prod.com:9092,kafka-2.prod.com:9092" client_id: "marmot-discovery" authentication: type: "sasl_ssl" username: "your-api-key" password: "your-api-secret" mechanism: "PLAIN" tls: enabled: true tags: - "kafka" - "streaming" ``` ## Development Build and test: ```sh make build make test ``` To run a local build inside Marmot: ```sh make install ``` This copies the binary to `~/.marmot/plugins/`, the directory Marmot scans for local plugins. A local plugin shadows the released core plugin with the same name: Marmot skips downloading it and loads your build instead. Delete the binary from `~/.marmot/plugins/` to fall back to the released version. If your Marmot runs with a custom plugins directory (`MARMOT_PLUGINS_DIR`), set the same value for `make install` so both point at the same place. ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | tags | multiselect | false | Tags to apply to discovered assets | | external_links | []object | false | External links to show on all assets | | external_links.name | string | true | Display name for the link | | external_links.icon | string | false | Icon identifier for the link | | external_links.url | string | true | URL to the external resource | | filter | object | false | Filter discovered assets by name (regex) | | filter.include | multiselect | false | Include patterns for resource names (regex) | | filter.exclude | multiselect | false | Exclude patterns for resource names (regex) | | bootstrap_servers | string | true | Comma-separated list of bootstrap servers | | client_id | string | false | Client ID for the consumer | | authentication | object | false | Authentication configuration | | authentication.type | select | false | Authentication type: none, sasl_plaintext, sasl_ssl, ssl | | authentication.username | string | false | SASL username | | authentication.password | password | false | SASL password | | authentication.mechanism | select | false | SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 | | consumer_config | string | false | Additional consumer configuration | | client_timeout_seconds | int | false | Request timeout in seconds | | tls | object | false | TLS configuration | | tls.enabled | bool | false | Whether to enable TLS | | tls.cert_path | string | false | Path to TLS certificate file | | tls.key_path | string | false | Path to TLS key file | | tls.ca_cert_path | string | false | Path to TLS CA certificate file | | tls.skip_verify | bool | false | Skip TLS verification | | schema_registry | object | false | Schema Registry configuration | | schema_registry.url | string | false | Schema Registry URL | | schema_registry.config | string | false | Additional Schema Registry configuration | | schema_registry.enabled | bool | false | Whether to use Schema Registry | | schema_registry.skip_verify | bool | false | Skip TLS certificate verification | | include_partition_info | bool | false | Whether to include partition information in metadata | | include_topic_config | bool | false | Whether to include topic configuration in metadata | ## Available Metadata ### Topic | Field | Type | Description | |-------|------|-------------| | topic_name | string | Name of the Kafka topic | | partition_count | int32 | Number of partitions | | replication_factor | int16 | Replication factor | | retention_ms | string | Message retention period in milliseconds | | retention_bytes | string | Maximum size of the topic in bytes | | cleanup_policy | string | Topic cleanup policy | | min_insync.replicas | string | Minimum number of in-sync replicas | | max_message.bytes | string | Maximum message size in bytes | | segment_bytes | string | Segment file size in bytes | | segment_ms | string | Segment file roll time in milliseconds | | delete_retention_ms | string | Time to retain deleted segments in milliseconds | | value_schema_id | int | ID of the value schema in Schema Registry | | value_schema_version | int | Version of the value schema | | value_schema_type | string | Type of the value schema (AVRO, JSON, etc.) | | value_schema | string | Value schema definition | | key_schema_id | int | ID of the key schema in Schema Registry | | key_schema_version | int | Version of the key schema | | key_schema_type | string | Type of the key schema (AVRO, JSON, etc.) | | key_schema | string | Key schema definition | --- ## Kubernetes Experimental Creates: AssetsLineageRun History The Kubernetes plugin discovers namespaces, services, deployments, stateful sets, cron jobs, and pods from Kubernetes clusters. Each resource kind can be toggled on or off, and discovery can be scoped to specific namespaces or a label selector. Discovered resources are linked together: namespaces contain their resources, services link to the deployments and stateful sets they expose (matched by selector), and workloads link to their pods (matched by owner references). Cron jobs come with run history built from their recent job runs, so the catalog shows whether the nightly pipeline actually succeeded. When `cluster_name` is set, a Cluster asset is created as the root of the tree. Pods are not discovered by default because they are short-lived and can flood the catalog; enable `discover_pods` when pod-level visibility is worth the churn. One-off Jobs are never cataloged for the same reason; only jobs owned by a cron job are used, as run history. ## Prerequisites The plugin needs read access to the resources it discovers. When running inside a cluster, bind a role like this to the service account Marmot runs as: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: marmot-discovery rules: - apiGroups: [""] resources: ["namespaces", "services", "pods"] verbs: ["get", "list"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "replicasets"] verbs: ["get", "list"] - apiGroups: ["batch"] resources: ["cronjobs", "jobs"] verbs: ["get", "list"] ``` :::tip[Managed clusters] This plugin is for self-managed and on-prem clusters. For managed clusters that authenticate with cloud IAM, use the dedicated plugins, which reuse this plugin's discovery engine: - Amazon EKS: the [EKS plugin](./EKS) - Google GKE: the [GKE plugin](./GKE) ::: :::tip[Authentication] The plugin supports three authentication methods: - In-cluster: when Marmot runs inside Kubernetes and no connection settings are provided, the pod's service account is used automatically. The projected token is rotated automatically, so there is nothing to refresh. - Kubeconfig: `$KUBECONFIG` or `~/.kube/config` is used when Marmot runs somewhere kubectl already works. Set `kubeconfig_path` and `context` to pick a specific file and context. - Direct token: set `host`, `token`, and `ca_certificate` to connect to any cluster with a service account token. ::: ### Connecting with a service account token Create a service account bound to the read-only role above and mint a token for it: ```bash kubectl create serviceaccount marmot-discovery kubectl create clusterrolebinding marmot-discovery \ --clusterrole=marmot-discovery --serviceaccount=default:marmot-discovery kubectl create token marmot-discovery --duration=48h ``` :::warning[Tokens expire] `kubectl create token` mints a time-bounded token, and the API server caps the lifetime (often 48h) regardless of the `--duration` you request, so a scheduled ingest will start failing once it expires. For unattended discovery, prefer in-cluster auth (its token is rotated automatically), or rotate the token on a schedule. A long-lived token can be created with a [`kubernetes.io/service-account-token` Secret](https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets), but that is discouraged upstream and disabled on some clusters. ::: Then give the plugin the cluster endpoint, the token, and the cluster's CA certificate. The connection fields go in the same config as the discovery options, not a separate file: ```yaml host: "https://mycluster.example.com:6443" token: "${K8S_SA_TOKEN}" ca_certificate: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- cluster_name: "prod" namespaces: - "payments" - "orders" discover_pods: false tags: - "kubernetes" - "${labels.team}" ``` ## Example Configuration When Marmot runs inside the cluster or has a working kubeconfig, no connection fields are needed; leave out `host`/`token` and the plugin uses the in-cluster service account or your kubeconfig. This example lists every discovery option with its default: ```yaml cluster_name: "prod" namespaces: - "payments" - "orders" discover_namespaces: true discover_services: true discover_deployments: true discover_statefulsets: true discover_cronjobs: true discover_pods: false labels_to_metadata: true annotations_to_metadata: false tags: - "kubernetes" - "${labels.team}" ``` Set `namespaces` to `["*"]` (or leave it empty) to discover all namespaces except the ones in `exclude_namespaces`. Tags interpolate resource labels, so `${labels.team}` tags every asset with the value of its `team` label. Set `cluster_name` when cataloging more than one cluster; it prefixes asset names (`prod/payments/api`) so the same namespace in two clusters stays distinct, and it creates a Cluster asset that anchors the lineage tree. ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | annotations_to_metadata | bool | false | Include resource annotations in asset metadata | | ca_certificate | string | false | PEM-encoded CA certificate of the API server | | cluster_name | string | false | Cluster name to prefix asset names with | | context | string | false | Kubeconfig context. Defaults to the current context | | host | string | false | API server URL for direct token authentication | | discover_cronjobs | bool | false | Discover cron jobs, with their recent job runs as run history | | discover_deployments | bool | false | Discover deployments | | discover_namespaces | bool | false | Discover namespaces | | discover_pods | bool | false | Discover pods. Off by default because pods are short-lived and can flood the catalog | | discover_services | bool | false | Discover services | | discover_statefulsets | bool | false | Discover stateful sets | | exclude_namespaces | []string | false | Namespaces to skip when discovering all namespaces | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | kubeconfig_path | string | false | Kubeconfig path. Defaults to in-cluster, then $KUBECONFIG | | label_selector | string | false | Only discover namespaced resources matching this label selector (e.g. team=data) | | labels_to_metadata | bool | false | Include resource labels in asset metadata | | namespaces | []string | false | Namespaces to discover. Empty or ["*"] means all namespaces | | tags | TagsConfig | false | Tags to apply to discovered assets | | token | string | false | Bearer token, typically a service account token | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | annotations | map[string]string | Resource annotations | | available_replicas | int32 | Number of available replicas | | cluster | string | Configured cluster name | | cluster_ip | string | Cluster IP address (None for headless services) | | concurrency_policy | string | Cron job concurrency policy (Allow, Forbid, Replace) | | container_count | int | Number of containers in the pod template | | created_at | string | Resource creation timestamp | | external_name | string | External DNS name for ExternalName services | | headless_service | string | Headless service governing the stateful set | | images | string | Container images (comma-separated) | | kubernetes_version | string | Kubernetes server version | | labels | map[string]string | Resource labels | | last_schedule_time | string | When the cron job last fired | | last_successful_time | string | When the cron job last completed successfully | | load_balancer_hosts | string | Load balancer ingress hostnames and IPs | | namespace | string | Namespace name | | node | string | Node the pod is scheduled on | | owner_kind | string | Kind of the controlling owner (ReplicaSet, StatefulSet, DaemonSet, Job) | | owner_name | string | Name of the controlling owner | | paused | bool | Whether rollouts are paused | | phase | string | Lifecycle phase (Active, Running, Pending, Failed) | | platform | string | Server platform (e.g. linux/amd64) | | ports | string | Exposed ports (name:port/protocol, comma-separated) | | qos_class | string | Quality of service class (Guaranteed, Burstable, BestEffort) | | ready_replicas | int32 | Number of ready replicas | | replicas | int32 | Desired replica count | | restart_count | int32 | Total container restarts | | schedule | string | Cron schedule expression | | selector | string | Pod selector labels (key=value, comma-separated) | | service_account | string | Service account the resource runs as | | service_type | string | Service type (ClusterIP, NodePort, LoadBalancer, ExternalName) | | strategy | string | Rollout/update strategy (RollingUpdate, Recreate, OnDelete) | | suspended | bool | Whether the cron job is suspended | | timezone | string | Time zone the cron schedule is evaluated in | | updated_replicas | int32 | Number of replicas updated to the latest pod template | | volume_claims | string | Volume claim templates (name:size/storageClass, comma-separated) | --- ## Lambda Experimental Creates: Assets The Lambda plugin discovers and catalogs AWS Lambda functions across your AWS accounts. It captures function metadata including runtime, memory, timeout, VPC configuration, layers, tracing, and tags. ## Required Permissions ## AWS Configuration See [AWS Configuration](./Shared%20Configuration/AWS%20Configuration.md) for the supported AWS configuration options. ## Example Configuration ```yaml credentials: region: "us-east-1" profile: "production" role: "" tags: - "aws" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials | AWSCredentials | false | AWS credentials configuration | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_tags | []string | false | List of AWS tags to include as metadata. By default, all tags are included. | | tags | TagsConfig | false | Tags to apply to discovered assets | | tags_to_metadata | bool | false | Convert AWS tags to Marmot metadata | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | architectures | string | Instruction set architectures (x86_64, arm64) | | code_sha256 | string | SHA256 hash of the deployment package | | code_size | int64 | The size of the function's deployment package in bytes | | description | string | The function's description | | environment_variable_count | int | Number of environment variables configured | | ephemeral_storage_mb | int32 | Ephemeral storage allocated in MB | | function_arn | string | The ARN of the Lambda function | | handler | string | The function's entry point handler | | last_modified | string | Date and time the function was last modified | | last_update_status | string | Status of the last update (Successful, Failed, InProgress) | | layer_count | int | Number of Lambda layers attached | | layers | string | Lambda layer ARNs attached to the function | | memory_size_mb | int32 | Memory allocated to the function in MB | | package_type | string | Deployment package type (Zip or Image) | | role | string | The IAM execution role ARN | | runtime | string | The runtime environment for the function (e.g. go1.x, python3.12, nodejs20.x) | | security_group_count | int | Number of VPC security groups | | state | string | Current state of the function (Active, Pending, Inactive, Failed) | | subnet_count | int | Number of VPC subnets | | tags | map[string]string | AWS resource tags | | timeout_seconds | int32 | Function execution timeout in seconds | | tracing_mode | string | X-Ray tracing mode (Active or PassThrough) | | version | string | The function version | | vpc_id | string | VPC ID if the function is connected to a VPC | --- ## MongoDB Experimental Creates: AssetsLineage The MongoDB plugin discovers databases and collections from MongoDB instances. It samples documents to infer schema and captures index information. ## Required Permissions The user needs read access to discover collections: ```javascript db.createUser({ user: "marmot_reader", pwd: "your-password", roles: [{ role: "read", db: "your_database" }] }) ``` For discovering all databases, use the `readAnyDatabase` role. ## Example Configuration ```yaml host: "mongo-cluster.company.com" port: 27017 user: "analytics_reader" password: "mongo_pass_456" auth_source: "admin" tls: true tags: - "mongodb" - "analytics" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | auth_source | string | false | Authentication database name | | connection_uri | string | false | MongoDB connection URI (overrides host/port/user/password) | | exclude_system_dbs | bool | false | Whether to exclude system databases (admin, config, local) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | MongoDB server hostname or IP address | | include_collections | bool | false | Whether to discover collections | | include_databases | bool | false | Whether to discover databases | | include_indexes | bool | false | Whether to include index information | | include_views | bool | false | Whether to include views | | password | string | false | Password for authentication | | port | int | false | MongoDB server port | | sample_schema | bool | false | Sample documents to infer schema | | sample_size | int | false | Number of documents to sample (-1 for entire collection) | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls | bool | false | Enable TLS/SSL for connection | | tls_insecure | bool | false | Skip verification of server certificate | | use_random_sampling | bool | false | Use random sampling for schema inference | | user | string | false | Username for authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | background | bool | Whether the index was built in the background | | capped | bool | Whether the collection is capped | | collection | string | Collection name | | created | string | Creation timestamp if available | | data_types | []string | Observed data types | | database | string | Database name | | description | string | Field description from validation schema if available | | document_count | int64 | Approximate document count | | field_name | string | Field name | | fields | string | Fields included in the index | | frequency | float64 | Frequency of field occurrence in documents | | host | string | MongoDB server hostname | | index_count | int | Number of indexes on collection | | is_required | bool | Whether field appears in all documents | | max_documents | int64 | Maximum document count for capped collections | | max_size | int64 | Maximum size for capped collections | | name | string | Index name | | object_type | string | Object type (collection, view) | | partial | bool | Whether the index is partial | | partial_filter | string | Filter expression for partial indexes | | port | int | MongoDB server port | | replicated | bool | Whether collection is replicated | | sample_values | string | Sample values from documents | | shard_key | string | Shard key if collection is sharded | | sharding_enabled | bool | Whether sharding is enabled | | size | int64 | Collection size in bytes | | sparse | bool | Whether the index is sparse | | storage_engine | string | Storage engine used | | ttl | int | Time-to-live in seconds if TTL index | | type | string | Index type (e.g., single field, compound, text, geo) | | unique | bool | Whether the index enforces uniqueness | | validation_action | string | Validation action if schema validation is enabled | | validation_level | string | Validation level if schema validation is enabled | --- ## MySQL Experimental Creates: AssetsLineage The MySQL plugin discovers databases and tables from MySQL instances. It captures column information, row counts, and foreign key relationships for lineage. ## Required Permissions The user needs read access to the information schema: ```sql CREATE USER 'marmot_reader'@'%' IDENTIFIED BY 'your-password'; GRANT SELECT ON your_database.* TO 'marmot_reader'@'%'; GRANT SELECT ON information_schema.* TO 'marmot_reader'@'%'; ``` ## Example Configuration ```yaml host: "mysql-prod.internal" port: 3306 user: "marmot_user" password: "mysql_secure_pass" database: "ecommerce" tls: "true" tags: - "mysql" - "ecommerce" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | database | string | false | Database name to connect to | | discover_foreign_keys | bool | false | Whether to discover foreign key relationships | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | MySQL server hostname or IP address | | include_columns | bool | false | Whether to include column information in table metadata | | include_row_counts | bool | false | Whether to include approximate row counts | | password | string | false | Password for authentication | | port | int | false | MySQL server port | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls | string | false | TLS configuration (false, true, skip-verify, preferred) | | user | string | false | Username for authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | character_set | string | Character set | | charset | string | Character set | | collation | string | Table collation | | collation | string | Collation | | column_default | string | Default value | | column_name | string | Column name | | column_type | string | Full column type definition | | comment | string | Object comment/description | | comment | string | Column comment/description | | constraint_name | string | Foreign key constraint name | | created | string | Creation timestamp | | data_length | int64 | Data size in bytes | | data_type | string | Data type | | database | string | Database name | | delete_rule | string | Delete rule (CASCADE, RESTRICT, etc.) | | engine | string | Storage engine | | host | string | MySQL server hostname | | index_length | int64 | Index size in bytes | | is_auto_increment | bool | Whether column auto-increments | | is_nullable | bool | Whether null values are allowed | | is_primary_key | bool | Whether column is part of primary key | | object_type | string | Object type (table, view) | | port | int | MySQL server port | | row_count | int64 | Approximate row count | | schema | string | Schema name | | source_column | string | Column in the referencing table | | source_schema | string | Schema of the referencing table | | source_table | string | Name of the referencing table | | table_name | string | Object name | | target_column | string | Column in the referenced table | | target_schema | string | Schema of the referenced table | | target_table | string | Name of the referenced table | | update_rule | string | Update rule (CASCADE, RESTRICT, etc.) | | updated | string | Last update timestamp | | version | string | MySQL version | --- ## NATS Experimental Creates: Assets The NATS plugin discovers JetStream streams from NATS servers. It connects using the NATS client protocol and enumerates streams via the JetStream API, collecting configuration and runtime state for each stream. ## Requirements - **JetStream must be enabled** on the NATS server (start with `-js` flag or configure in `nats-server.conf`). Core NATS subjects are ephemeral and not discoverable as persistent assets. - The connecting user needs permission to access the JetStream API (`$JS.API.>`). ## Authentication The plugin supports several authentication methods: - **Token**: Set the `token` field for token-based auth. - **Username/Password**: Set `username` and `password` fields. - **Credentials file**: Set `credentials_file` to the path of a `.creds` file (NKey-based auth). - **TLS**: Enable `tls` for encrypted connections. Use `tls_insecure` to skip certificate verification in development. ## Example Configuration ```yaml host: "localhost" port: 4222 token: "s3cr3t" filter: include: - "^ORDERS" tags: - "nats" - "messaging" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials_file | string | false | Path to NATS credentials file (.creds) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | NATS server hostname or IP address | | password | string | false | Password for authentication | | port | int | false | NATS server port | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls | bool | false | Enable TLS connection | | tls_insecure | bool | false | Skip TLS certificate verification | | token | string | false | Authentication token | | username | string | false | Username for authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | bytes | uint64 | Total bytes stored in the stream | | consumer_count | int | Number of consumers attached to the stream | | discard_policy | string | Policy when limits are reached (Old or New) | | duplicate_window | string | Duplicate message tracking window | | first_seq | uint64 | Sequence number of the first message | | host | string | NATS server hostname | | last_seq | uint64 | Sequence number of the last message | | max_age | string | Maximum age of messages | | max_bytes | int64 | Maximum total bytes for the stream (-1 = unlimited) | | max_msg_size | int64 | Maximum size of a single message | | max_msgs | int64 | Maximum number of messages (-1 = unlimited) | | messages | uint64 | Total number of messages in the stream | | num_replicas | int | Number of stream replicas | | port | int | NATS server port | | retention_policy | string | Message retention policy (Limits, Interest, WorkQueue) | | storage_type | string | Storage backend (File or Memory) | | stream_name | string | Name of the JetStream stream | | subjects | string | Comma-separated list of subjects the stream listens on | --- ## OpenAPI Experimental Creates: Assets The OpenAPI plugin discovers API specifications from OpenAPI v3 files. It creates assets for services and their endpoints. The plugin scans for `.json` and `.yaml` files and parses them as OpenAPI v3 specifications. ## File Sources The `spec_path` field accepts local paths, S3 URIs (`s3://bucket/prefix`) or Git URIs (`git::https://...`). For S3 and Git sources, files are downloaded to a temporary directory before discovery and cleaned up afterwards. See [File Sources](./Shared%20Configuration/File%20Sources.md) for the full list of supported backends, authentication options and configuration examples. ## Example Configuration ```yaml spec_path: "/app/openapi-specs" tags: - "openapi" - "specifications" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | git_source | GitSourceConfig | false | Git repository file source configuration | | s3_source | S3SourceConfig | false | S3 file source configuration | | source_type | string | false | File source backend (auto-detected from path when empty) | | spec_path | string | false | Path to the directory containing the OpenAPI specifications (local path, s3://bucket/prefix or git::url) | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | contact_email | string | Contact email | | contact_name | string | Contact name | | contact_url | string | Contact URL | | deprecated | bool | Is this endpoint deprecated | | description | string | Description of the API | | description | string | A verbose explanation of the operation behaviour. | | external_docs | string | Link to the external documentation | | http_method | string | HTTP method | | license_identifier | string | SPDX license experession for the API | | license_name | string | Name of the license | | license_url | string | URL of the license | | num_deprecated_endpoints | int | Number of deprecated endpoints in the OpenAPI specification | | num_endpoints | int | Number of endpoints in the OpenAPI specification | | openapi_version | string | Version of the OpenAPI spec | | operation_id | string | Unique identifier of the operation | | path | string | Path | | servers | []string | URL of the servers of the API | | service_name | string | Name of the service that owns the resource | | service_version | string | Version of the service | | status_codes | []string | All HTTP response status codes that are returned for this endpoint. | | summary | string | A short summary of what the operation does | | terms_of_service | string | Link to the page that describes the terms of service | --- ## OpenMetadata Experimental Creates: AssetsLineageRun HistoryGlossary The OpenMetadata plugin imports an entire OpenMetadata instance in one run: tables, views, stored procedures, topics, buckets, dashboards, charts, pipelines, models, search indices, API endpoints, drive files and spreadsheets, the business glossary, and the lineage between them. OpenMetadata is a catalog, so everything in it describes something that lives somewhere else. The plugin catalogues each entity as the technology it actually describes rather than as an OpenMetadata thing: a table under a Postgres service becomes a PostgreSQL asset in Marmot, addressed exactly as Marmot's own PostgreSQL plugin would address it. Technologies Marmot has no plugin for yet, such as Snowflake or Looker, come across under their own provider name. Point Marmot at OpenMetadata and the result looks like a catalog Marmot built itself. OpenMetadata 1.4 or newer is supported. The plugin negotiates the field list with the server on the first call to each endpoint, so one plugin binary works across every version in that range, and entity kinds a server predates (drives, API collections, and so on) are skipped without failing the run. ## What it imports | OpenMetadata | Marmot asset type | |---|---| | database or schema, whichever the engine really has | Database, Dataset, Namespace or Catalog | | table | Table, View, or the name the technology's own plugin uses (a MongoDB collection is a Collection, a BigQuery external table an ExternalTable) | | stored procedure | Function | | topic | Topic | | container | Bucket, or Container on Azure Blob. Only the top level one is imported | | drive | Drive | | drive folder | Folder | | drive file | File | | spreadsheet | Spreadsheet | | a sheet of a spreadsheet | Table, with its columns | | dashboard, chart, dashboard data model | Dashboard, Chart, Data Model Object | | pipeline and its tasks | Pipeline, Task | | ML model | Model | | search index | Table on Elasticsearch and OpenSearch, Index elsewhere | | API collection and endpoint | Service, Endpoint | | glossary | Glossary Term, the root of its own vocabulary | | glossary term | Glossary Term, nested under its parent term or its glossary | | entity lineage | Lineage, carrying the pipeline that moved the data | | pipeline executions | Run History | Descriptions, columns, classification tags, owners, domains and data products come across on each asset, and nothing loses its trail: every asset gets an OpenMetadata link that jumps straight to the entity it was imported from, and an `openmetadata` metadata object carrying the fully qualified name, the service and when the entity last changed. Mid-migration, any asset in Marmot traces back to its source in one click. The glossary comes across as glossary terms rather than as tags, so a term keeps its definition, its synonyms and the terms below it, and the assets it was curated onto are assigned it. Terms are identified by their OpenMetadata fully qualified name, so two glossaries can each hold a `Customer` without becoming one term. Set `glossary_terms_as_tags: true` to also copy each assigned term onto the asset's tags, or `include_glossary: false` to leave the glossary behind entirely. Object storage comes across as the bucket alone. OpenMetadata models the prefixes inside a bucket as containers of their own, but Marmot's S3, GCS and Azure Blob plugins catalogue the bucket and nothing below it, so an imported prefix would sit in the catalog forever without a native run ever updating it. Each run reports how many it left out. Set `include_container_prefixes: true` to import the hierarchy anyway, which is worth doing when nothing else is going to catalogue that bucket. Drives are different and come across in full: a drive really is a tree of folders, and Marmot's GoogleDrive plugin catalogues it as one. The drive itself is catalogued, folders are linked up to its root, and documents are placed by their path rather than their OpenMetadata name, because OpenMetadata files some of them under the service instead of the folder they live in. Folders that exist only in a path are created too, marked `inferred_from_path`, so nothing dangles. Columns are part of the asset in Marmot rather than entities of their own, so a table with two hundred columns is one asset carrying two hundred columns, not two hundred and one things. That is why OpenMetadata's own totals are far larger than the number of assets an import produces. Marmot ingestion runs cannot create teams, users, domains or data products as objects of their own, so those stay on the assets as metadata. Data quality test cases and OpenMetadata's own usage analytics have no Marmot equivalent and are not imported. ## The Migration A catalog migration fails when it has to happen all at once. This plugin is built to run on a schedule for as long as the move takes, so no single day has to be the day everything switches. **1. Schedule the import.** Set it up as a recurring pipeline through the UI wizard, the CLI, Terraform, Pulumi or the REST API; the [Populating docs](/docs/Populating/) cover each. Everything is imported by default, and the configuration below covers scoping down to specific services or service types. **2. Keep working in both catalogs.** Each run brings across whatever changed in OpenMetadata, so the two stay in step while people are still working in both. Re-running is safe: assets that have not changed are left alone. Anything written in Marmot survives every re-sync, because a description edited in Marmot is stored separately from the imported one, so the next sync refreshes the imported side and never overwrites the edit. The same holds for tags, owners and glossary terms added in Marmot. **3. Adopt native plugins one system at a time.** When you are ready to catalogue a system directly, add its own pipeline, for example the [PostgreSQL plugin](/docs/Plugins/PostgreSQL) against the database OpenMetadata was describing. Imported and native assets share an identity, so the native run takes over the assets that are already there instead of creating a second copy. Nothing needs to be deleted or re-pointed, and the descriptions people wrote stay put. **4. Switch OpenMetadata off.** When nothing depends on it anymore, stop scheduling the run. The imported assets stay exactly as they are. ## Running it Alongside Marmot's Own Plugins By default an imported asset lands on the same MRN the technology's native Marmot plugin would use, so the two runs contribute to one asset instead of creating two. A Postgres table becomes `mrn://table/postgresql/orders` whether Marmot read it from OpenMetadata or from the database itself, so whichever run happens next updates the asset that is already there. That means names drop the levels the native plugin does not use. Marmot's own plugins for Postgres, MySQL, BigQuery, MongoDB, ClickHouse, Glue and Iceberg all name a table by its bare name, so `public.orders` and `staging.orders` resolve to one asset, as do two OpenMetadata services holding the same table name. The run reports every entity it merged this way. Set `naming: qualified` to keep them apart instead, at the cost of no longer merging with native runs, which also gives up the handover described above: ```yaml runs: - openmetadata: host: "https://openmetadata.company.com" jwt_token: "eyJraWQiOiJHYjM4OWEtOWY3Ni1nZGpzLWE5..." naming: qualified ``` ## Getting a Token The plugin authenticates as a bot or as a user, with a JWT. For a bot, open **Settings → Bots** in OpenMetadata, pick a bot such as `ingestion-bot`, and copy its token. For a user, open **Settings → Members**, pick the user, and create a personal access token. The token needs read access to the entities you want to import. ## Example Configuration ```yaml host: "https://openmetadata.company.com" jwt_token: "eyJraWQiOiJHYjM4OWEtOWY3Ni1nZGpzLWE5..." exclude_service_types: - "Metadata" tags: - "openmetadata" ``` Import a single service: ```yaml host: "https://openmetadata.company.com" jwt_token: "eyJraWQiOiJHYjM4OWEtOWY3Ni1nZGpzLWE5..." services: - "postgres_prod" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | concurrency | int | false | Parallel lineage requests | | exclude_service_types | []string | false | OpenMetadata service types to skip | | exclude_services | []string | false | OpenMetadata services to skip | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | glossary_terms_as_tags | bool | false | Also copy assigned glossary terms onto assets as tags. They are imported as glossary terms either way | | host | string | true | OpenMetadata server URL, for example https://openmetadata.company.com | | include_apis | bool | false | Import API collections and endpoints | | include_columns | bool | false | Import column, field and feature definitions | | include_container_prefixes | bool | false | Also import the prefixes and folders inside a storage container. Marmot's own object storage plugins catalogue only the container itself | | include_containers | bool | false | Import object storage buckets and containers | | include_drives | bool | false | Import drive directories, files, spreadsheets and worksheets | | include_dashboards | bool | false | Import dashboards, charts and dashboard data models | | include_deleted | bool | false | Import entities OpenMetadata has soft deleted | | include_glossary | bool | false | Import the business glossary as Marmot glossary terms, and assign them to the assets they are curated onto | | include_lineage | bool | false | Import lineage between imported assets | | include_mlmodels | bool | false | Import machine learning models | | include_pipelines | bool | false | Import orchestration pipelines | | include_run_history | bool | false | Import recent pipeline executions as run history | | include_search_indexes | bool | false | Import search indices | | include_stored_procedures | bool | false | Import stored procedures as functions | | include_tables | bool | false | Import databases, tables and views | | include_tasks | bool | false | Import the individual tasks of each pipeline | | include_topics | bool | false | Import messaging topics | | insecure_skip_verify | bool | false | Skip TLS certificate verification | | jwt_token | string | true | Bot token or personal access token from OpenMetadata | | link_to_openmetadata | bool | false | Add a link back to the entity in OpenMetadata on every asset | | naming | select | false | native names assets the way Marmot's own plugin for each technology names them, so a later native run merges with the imported assets. qualified uses the full OpenMetadata path, which keeps two services of the same technology apart | | page_size | int | false | Entities per API request | | run_history_days | int | false | How many days of pipeline executions to import | | run_history_limit | int | false | Maximum executions to import per pipeline | | service_types | []string | false | Only import these OpenMetadata service types, for example Postgres or Kafka (all if empty) | | services | []string | false | Only import these OpenMetadata services (all if empty) | | source_priority | int | false | Priority of OpenMetadata against other sources of the same asset. Lower wins | | tags | []string | false | Tags to apply to discovered assets | | tags_from_openmetadata | bool | false | Copy OpenMetadata classification tags onto assets | | timeout_seconds | int | false | Per-request timeout | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | algorithm | string | Algorithm the model uses | | bucket | string | Top level container the object lives in | | chart_count | int | Number of charts on the dashboard | | chart_type | string | Chart type reported by the BI tool | | cleanup_policies | []string | Topic cleanup policies | | collection | string | API collection the endpoint belongs to | | column_count | int | Number of columns | | concurrency | int | Maximum concurrent runs | | dashboard_type | string | Dashboard type reported by the BI tool | | data_model_type | string | Data model type reported by the BI tool | | data_products | []string | OpenMetadata data products the entity belongs to | | database | string | Database name | | domains | []string | OpenMetadata domains the entity belongs to | | downstream_tasks | []string | Tasks that run after this one | | endpoint_url | string | URL of the endpoint | | feature_count | int | Number of features | | field_count | int | Number of fields in the index | | file_formats | []string | File formats found in the container | | glossary_terms | []string | Glossary terms assigned to the entity | | image_repository | string | Repository holding the model image | | index_type | string | Index type | | max_message_size | int | Maximum message size in bytes | | method | string | HTTP method | | object_count | int64 | Number of objects | | object_type | string | OpenMetadata table type, for example Regular, View or MaterializedView | | openmetadata.fqn | string | Fully qualified name of the entity in OpenMetadata | | openmetadata.id | string | OpenMetadata entity id | | openmetadata.service | string | OpenMetadata service the entity belongs to | | openmetadata.service_type | string | OpenMetadata service type, for example Postgres or Looker | | openmetadata.updated_at | string | When the entity last changed in OpenMetadata | | openmetadata.url | string | Address of the entity in the OpenMetadata UI | | owners | []string | Users or teams that own the entity in OpenMetadata | | partitioned | bool | Whether the container is partitioned | | partitions | int | Number of partitions | | path | string | Request path | | pipeline | string | Pipeline a task belongs to | | prefix | string | Path prefix within the bucket | | primary_key | []string | Columns forming the primary key | | procedure_type | string | Stored procedure type | | project | string | Project or workspace the dashboard belongs to | | replication_factor | int | Replication factor | | retention_ms | int64 | Retention time in milliseconds | | retention_size | int64 | Retention size in bytes | | row_count | int64 | Row count from the OpenMetadata profiler | | schedule_interval | string | Schedule the pipeline runs on | | schema | string | Schema name | | schema_type | string | Message schema type, for example Avro or JSON | | server | string | Address the model is served from | | size | int64 | Size in bytes | | storage | string | Where the model artefact is stored | | table_name | string | Object name | | target | string | Column the model predicts | | task_count | int | Number of tasks in the pipeline | | task_type | string | Task type, for example the Airflow operator | | shared | bool | Whether the drive directory or file is shared | | file_type | string | Drive file type, for example Document or Spreadsheet | | file_extension | string | Drive file extension | | mime_type | string | Drive file MIME type | | directory_type | string | Drive directory type | | path | string | Path within the drive | | weekly_query_count | int | Queries against the table in the last week | --- ## OpenSearch Experimental Creates: AssetsLineage The OpenSearch plugin discovers indices, data streams and aliases from OpenSearch clusters. ## Required Permissions The connecting user needs `cluster_monitor` and `indices_monitor` permissions. The built-in `readall` role is usually sufficient. ## Example Configuration ```yaml addresses: - "https://opensearch.company.com:9200" username: "admin" password: "admin" tags: - "opensearch" - "search" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | addresses | []string | false | List of OpenSearch node URLs | | ca_cert_path | string | false | Path to a custom CA certificate file | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_aliases | bool | false | Discover aliases | | include_data_streams | bool | false | Discover data streams | | include_index_stats | bool | false | Collect document count and store size metrics | | include_system_indices | bool | false | Include system indices (prefixed with .) | | password | string | false | Password for basic authentication | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls_skip_verify | bool | false | Skip TLS certificate verification | | username | string | false | Username for basic authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | alias_name | string | Name of the alias | | analyzer | string | Analyzer used for the field | | backing_indices | int | Number of backing indices | | cluster | string | Name of the OpenSearch cluster | | creation_date | string | Date and time when the index was created | | data_stream_name | string | Name of the data stream | | docs_count | int64 | Number of documents in the index | | field_name | string | Full dotted path of the field | | field_type | string | OpenSearch field type (keyword, text, long, etc.) | | filter_defined | string | Whether a filter is defined on the alias | | generation | int | Current generation of the data stream | | health | string | Health status of the index (green, yellow, red) | | index | string | Whether the field is indexed | | index_name | string | Name of the index | | indices | string | Comma-separated list of indices the alias points to | | is_write_index | string | Whether the alias has a designated write index | | replicas | int | Number of replica shards | | shards | int | Number of primary shards | | status | string | Health status of the data stream | | status | string | Open/close status of the index | | store_size | string | Total store size of the index | | template | string | Index template used by the data stream | | timestamp_field | string | Name of the timestamp field | | uuid | string | UUID of the index | --- ## PostgreSQL Experimental Creates: AssetsLineage The PostgreSQL plugin discovers databases, schemas, and tables from PostgreSQL instances. It captures column information, table metrics, and foreign key relationships for lineage. ## Required Permissions The user needs read access to the information schema: ```sql GRANT CONNECT ON DATABASE your_db TO marmot_reader; GRANT USAGE ON SCHEMA public TO marmot_reader; GRANT SELECT ON ALL TABLES IN SCHEMA public TO marmot_reader; ``` ## Example Configuration ```yaml host: "prod-postgres.company.com" port: 5432 user: "marmot_reader" password: "secure_password_123" ssl_mode: "require" tags: - "postgres" - "production" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | discover_foreign_keys | bool | false | Whether to discover foreign key relationships | | enable_metrics | bool | false | Whether to include table metrics | | exclude_system_schemas | bool | false | Whether to exclude system schemas (pg_*) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | PostgreSQL server hostname or IP address | | include_columns | bool | false | Whether to include column information in table metadata | | include_databases | bool | false | Whether to discover databases | | password | string | false | Password for authentication | | port | int | false | PostgreSQL server port | | ssl_mode | string | false | SSL mode (disable, require, verify-ca, verify-full) | | tags | TagsConfig | false | Tags to apply to discovered assets | | user | string | false | Username for authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | allow_connections | bool | Whether connections to this database are allowed | | collate | string | Database collation | | column_default | string | Default value expression | | column_name | string | Column name | | comment | string | Column comment/description | | comment | string | Object comment/description | | connection_limit | int | Maximum allowed connections | | constraint_name | string | Foreign key constraint name | | created | string | Creation timestamp | | ctype | string | Database character classification | | data_type | string | Data type | | database | string | Database name | | encoding | string | Database encoding | | host | string | PostgreSQL server hostname | | is_nullable | bool | Whether null values are allowed | | is_primary_key | bool | Whether column is part of primary key | | is_template | bool | Whether database is a template | | object_type | string | Object type (table, view, materialized_view) | | owner | string | Object owner | | port | int | PostgreSQL server port | | row_count | int64 | Approximate row count | | schema | string | Schema name | | size | int64 | Object size in bytes | | source_column | string | Column in the referencing table | | source_schema | string | Schema of the referencing table | | source_table | string | Name of the referencing table | | table_name | string | Object name | | target_column | string | Column in the referenced table | | target_schema | string | Schema of the referenced table | | target_table | string | Name of the referenced table | --- ## Redis Experimental Creates: Assets The Redis plugin discovers logical databases (db0–db15) from Redis instances. It uses the `INFO` command to collect server metadata and parses the Keyspace section to identify databases that contain keys. ## Required Permissions The connecting user needs permission to run the `INFO` command. By default all users can run `INFO`, but if you are using Redis ACLs: ``` ACL SETUSER marmot_reader on >password ~* &* +info +ping +select ``` ## Example Configuration ```yaml host: "localhost" port: 6379 password: "secret" discover_all_databases: true filter: include: - "^db[0-3]$" tags: - "redis" - "cache" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | db | int | false | Default database number | | discover_all_databases | bool | false | Discover all databases with keys (db0-db15) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | Redis server hostname or IP address | | password | string | false | Password for authentication | | port | int | false | Redis server port | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls | bool | false | Enable TLS connection | | tls_insecure | bool | false | Skip TLS certificate verification | | username | string | false | Username for ACL authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | avg_ttl_ms | int64 | Average TTL in milliseconds | | connected_clients | string | Number of connected clients | | database | string | Database name (e.g. db0) | | expires_count | int64 | Number of keys with an expiration | | host | string | Redis server hostname | | key_count | int64 | Number of keys in the database | | maxmemory_policy | string | Eviction policy when maxmemory is reached | | port | int | Redis server port | | redis_version | string | Redis server version | | role | string | Replication role (master/slave) | | uptime_seconds | string | Server uptime in seconds | | used_memory_human | string | Human-readable used memory | --- ## Redpanda Experimental Creates: Assets The Redpanda plugin discovers topics from Redpanda clusters. It uses the same discovery engine as the Kafka plugin since Redpanda is Kafka API-compatible. Because it is the same engine, topics are catalogued under the Kafka provider and addressed as `mrn://topic/kafka/`, not under a Redpanda provider. Running Kafka, Redpanda and Confluent Cloud against clusters that share a topic name will therefore land them on one asset. ## Connection ### Redpanda Cloud ```yaml bootstrap_servers: "seed-xxxxx.cloud.redpanda.com:9092" client_id: "marmot-discovery" authentication: type: "sasl_ssl" username: "your-username" password: "your-password" mechanism: "SCRAM-SHA-256" tls: enabled: true ``` ### Self-Hosted Redpanda ```yaml bootstrap_servers: "redpanda-0.example.com:9092,redpanda-1.example.com:9092" client_id: "marmot-discovery" ``` ## Example Configuration ```yaml bootstrap_servers: "kafka-1.prod.com:9092,kafka-2.prod.com:9092" client_id: "marmot-discovery" authentication: type: "sasl_ssl" username: "your-api-key" password: "your-api-secret" mechanism: "PLAIN" tls: enabled: true tags: - "kafka" - "streaming" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | authentication | AuthConfig | false | Authentication configuration | | bootstrap_servers | string | false | Comma-separated list of bootstrap servers | | client_id | string | false | Client ID for the consumer | | client_timeout_seconds | int | false | Request timeout in seconds | | consumer_config | map[string]string | false | Additional consumer configuration | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_partition_info | bool | false | Whether to include partition information in metadata | | include_topic_config | bool | false | Whether to include topic configuration in metadata | | schema_registry | SchemaRegistryConfig | false | Schema Registry configuration | | tags | TagsConfig | false | Tags to apply to discovered assets | | tls | TLSConfig | false | TLS configuration | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | cleanup_policy | string | Topic cleanup policy | | delete_retention_ms | string | Time to retain deleted segments in milliseconds | | key_schema | string | Key schema definition | | key_schema_id | int | ID of the key schema in Schema Registry | | key_schema_type | string | Type of the key schema (AVRO, JSON, etc.) | | key_schema_version | int | Version of the key schema | | max_message_bytes | string | Maximum message size in bytes | | min_insync_replicas | string | Minimum number of in-sync replicas | | partition_count | int32 | Number of partitions | | replication_factor | int16 | Replication factor | | retention_bytes | string | Maximum size of the topic in bytes | | retention_ms | string | Message retention period in milliseconds | | segment_bytes | string | Segment file size in bytes | | segment_ms | string | Segment file roll time in milliseconds | | topic_name | string | Name of the Kafka topic | | value_schema | string | Value schema definition | | value_schema_id | int | ID of the value schema in Schema Registry | | value_schema_type | string | Type of the value schema (AVRO, JSON, etc.) | | value_schema_version | int | Version of the value schema | --- ## S3 Experimental Creates: Assets The S3 plugin discovers and catalogs Amazon S3 buckets across your AWS accounts. It captures bucket metadata including security configurations, lifecycle policies, encryption settings, and tags. ## Required Permissions ## AWS Configuration See [AWS Configuration](./Shared%20Configuration/AWS%20Configuration.md) for the supported AWS configuration options. ## Example Configuration ```yaml credentials: region: "us-east-1" id: "" secret: "" tags: - "s3" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials | AWSCredentials | false | AWS credentials configuration | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_tags | []string | false | List of AWS tags to include as metadata. By default, all tags are included. | | tags | TagsConfig | false | Tags to apply to discovered assets | | tags_to_metadata | bool | false | Convert AWS tags to Marmot metadata | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | accelerate_config | string | Transfer acceleration configuration | | bucket_arn | string | The ARN of the S3 bucket | | creation_date | string | When the bucket was created | | encryption | string | Bucket encryption configuration | | lifecycle_config | string | Bucket lifecycle configuration | | logging_config | string | Bucket access logging configuration | | notification_config | string | Bucket notification configuration | | public_access_block | string | Public access block configuration | | region | string | The AWS region where the bucket is located | | replication_config | string | Bucket replication configuration | | request_payment_config | string | Request payment configuration | | tags | map[string]string | AWS resource tags | | versioning | string | Bucket versioning status | | website_config | string | Static website hosting configuration | --- ## SNS Experimental Creates: Assets The SNS plugin discovers and catalogs Amazon SNS topics across your AWS accounts. It captures topic configurations, subscription details, and tags. ## Required Permissions ## AWS Configuration See [AWS Configuration](./Shared%20Configuration/AWS%20Configuration.md) for the supported AWS configuration options. ## Example Configuration ```yaml credentials: region: "us-east-1" profile: "production" role: "" tags: - "aws" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials | AWSCredentials | false | AWS credentials configuration | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_tags | []string | false | List of AWS tags to include as metadata. By default, all tags are included. | | tags | TagsConfig | false | Tags to apply to discovered assets | | tags_to_metadata | bool | false | Convert AWS tags to Marmot metadata | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | display_name | string | Display name of the topic | | owner | string | AWS account ID that owns the topic | | policy | string | Access policy of the topic | | subscriptions_confirmed | string | Number of confirmed subscriptions | | subscriptions_pending | string | Number of pending subscriptions | | tags | map[string]string | AWS resource tags | | topic_arn | string | The ARN of the SNS topic | --- ## SQLite Experimental Creates: AssetsLineage The SQLite plugin discovers tables, views and foreign key relationships from SQLite database files. It opens the file read-only with the pure-Go `modernc.org/sqlite` driver, so it needs no cgo. Turso and libSQL databases use the SQLite on-disk format, so a local copy of one is discovered the same way as any other SQLite file. ## File Sources The `path` field accepts local paths, S3 URIs (`s3://bucket/key`) or Git URIs (`git::https://...`). For S3 and Git sources, the file is downloaded to a temporary directory before discovery and cleaned up afterwards. See [File Sources](./Shared%20Configuration/File%20Sources.md) for the full list of supported backends, authentication options and configuration examples. ## Example Configuration ```yaml path: "/data/app.db" include_columns: true enable_metrics: true discover_foreign_keys: true exclude_system_tables: true filter: include: - "^user.*" exclude: - ".*_tmp$" tags: - "sqlite" - "app" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | discover_foreign_keys | bool | false | Whether to discover foreign key relationships | | enable_metrics | bool | false | Whether to include table metrics (row and column counts) | | exclude_system_tables | bool | false | Whether to exclude SQLite internal tables (sqlite_*) | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | git_source | GitSourceConfig | false | Git repository file source configuration | | include_columns | bool | false | Whether to include column information in table metadata | | path | string | false | Path to the SQLite database file (local path, s3://bucket/key or git::url) | | s3_source | S3SourceConfig | false | S3 file source configuration | | source_type | string | false | File source backend (auto-detected from path when empty) | | tags | TagsConfig | false | Tags to apply to discovered assets | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | column_default | string | Default value expression | | column_name | string | Column name | | data_type | string | Declared column data type | | is_nullable | bool | Whether null values are allowed | | is_primary_key | bool | Whether the column is part of the primary key | | object_type | string | Object type (table, view) | | path | string | Path to the SQLite database file | | source_column | string | Column in the referencing table | | source_table | string | Name of the referencing table | | table_name | string | Table or view name | | target_column | string | Column in the referenced table | | target_table | string | Name of the referenced table | --- ## SQS Experimental Creates: AssetsLineage The SQS plugin discovers and catalogs Amazon SQS queues across your AWS accounts. It captures queue configurations and can discover Dead Letter Queue relationships. ## Required Permissions ## AWS Configuration See [AWS Configuration](./Shared%20Configuration/AWS%20Configuration.md) for the supported AWS configuration options. ## Example Configuration ```yaml credentials: region: "us-east-1" id: "" secret: "" tags: - "sns" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | credentials | AWSCredentials | false | AWS credentials configuration | | discover_dlq | bool | false | Discover Dead Letter Queue relationships | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | include_tags | []string | false | List of AWS tags to include as metadata. By default, all tags are included. | | tags | TagsConfig | false | Tags to apply to discovered assets | | tags_to_metadata | bool | false | Convert AWS tags to Marmot metadata | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | content_based_deduplication | bool | Whether content-based deduplication is enabled | | deduplication_scope | string | Deduplication scope for FIFO queues | | delay_seconds | string | Delay seconds for messages | | fifo_queue | bool | Whether this is a FIFO queue | | fifo_throughput_limit | string | FIFO throughput limit type | | maximum_message_size | string | Maximum message size in bytes | | message_retention_period | string | Message retention period in seconds | | queue_arn | string | The ARN of the SQS queue | | receive_message_wait_time | string | Long polling wait time in seconds | | redrive_policy | string | Redrive policy JSON string | | tags | map[string]string | AWS resource tags | | visibility_timeout | string | The visibility timeout for the queue | --- ## AWS Configuration AWS-backed plugins such as [S3](../S3.md), [SQS](../SQS.md), [SNS](../SNS.md), [Glue](../Glue.md), [Lambda](../Lambda.md), [DynamoDB](../DynamoDB.md) share common credentials configuration. The same config is used by the S3 backend in [File Sources](./File%20Sources.md). ## Credentials Credentials are configured under `credentials`: ```yaml credentials: use_default: true region: "us-east-1" ``` | Field | Description | | ------------------ | ---------------------------------------------------------------------------------------------- | | `use_default` | Use the default AWS credential chain (env vars, shared config, instance profile). Recommended. | | `id` | AWS access key ID. | | `secret` | AWS secret access key. | | `token` | AWS session token. | | `profile` | Profile name from the shared credentials file. | | `role` | IAM role ARN to assume. | | `role_external_id` | External ID for cross-account role assumption. | | `region` | AWS region. | | `endpoint` | Custom endpoint URL (useful for LocalStack or other S3-compatible services). | ## Default credential chain The recommended option. Marmot will pick up credentials from environment variables, `~/.aws/credentials`, `~/.aws/config`, container credentials or EC2/EKS instance profiles in the usual order. ```yaml credentials: use_default: true region: "us-east-1" ``` ## Static credentials Provide an access key pair directly. Use a session `token` for temporary credentials. ```yaml credentials: id: "AKIA..." secret: "..." region: "us-east-1" ``` ## Named profile Use a named profile from `~/.aws/credentials` or `~/.aws/config`. ```yaml credentials: profile: "my-profile" region: "us-east-1" ``` ## Assume role Assume an IAM role after loading the base credentials. `role_external_id` is supported for cross-account assumption. ```yaml credentials: use_default: true role: "arn:aws:iam::123456789012:role/MarmotReader" role_external_id: "optional-external-id" region: "us-east-1" ``` ## Custom endpoint Point at a non-AWS endpoint such as LocalStack. ```yaml credentials: use_default: true region: "us-east-1" endpoint: "http://localhost:4566" ``` --- ## File Sources Some plugins such as [DBT](../DBT.md), [DuckDB](../DuckDB.md) and [OpenAPI](../OpenAPI.md) read their input from files. Wherever a plugin exposes a path field (`path`, `target_path`, `spec_path`) it can point at one of three backends: | Backend | Description | |---------|-------------| | `local` | A path on the machine running Marmot. | | `s3` | An object or prefix in S3. Downloaded to a temp directory before discovery. | | `git` | A path inside a Git repo. Shallow-cloned to a temp directory before discovery. | The backend is auto-detected from the path prefix, or you can set it explicitly with `source_type`. Temporary files are cleaned up after the plugin runs. ## Local The default. Any path that is not an `s3://` or `git::` URI is treated as a local filesystem path. ```yaml path: "/data/analytics.duckdb" ``` ## S3 Use `s3://bucket/key` for a single file or `s3://bucket/prefix/` for a directory. ```yaml path: "s3://my-bucket/databases/analytics.duckdb" s3_source: credentials: region: "us-east-1" use_default: true ``` Set `source_type: "s3"` explicitly if your path does not start with `s3://`: ```yaml source_type: "s3" spec_path: "openapi/" s3_source: bucket: "api-specs" prefix: "openapi/" credentials: region: "us-east-1" use_default: true ``` `s3_source.credentials` accepts the same fields as other AWS-backed plugins. See [AWS Configuration](<./AWS Configuration.md>) for the full list. ## Git Use `git::`, optionally with a subpath (`//subdir`) and a `?ref=` query parameter. ```yaml path: "git::https://github.com/org/repo//data/analytics.duckdb?ref=main" ``` Or configure it explicitly: ```yaml source_type: "git" target_path: "target" git_source: url: "https://github.com/org/dbt-project" ref: "main" path: "target" token: "ghp_xxxx" ``` Authentication options: | Field | Description | |----------------|-------------| | `token` | Personal access token for HTTPS auth. | | `ssh_key_path` | Path to an SSH private key for SSH auth. | Public repos can be cloned anonymously. `ref` defaults to `main` and may be a branch or a tag. --- ## Trino Experimental Creates: AssetsLineage The Trino plugin discovers all catalogs (connected data sources like PostgreSQL, Hive, Iceberg, S3, etc.), their schemas, and tables/views. ## Required Permissions The connecting user needs `SELECT` access to `system.metadata.catalogs`, `system.metadata.table_comments`, and each catalog's `information_schema`. A read-only user with access to these system tables is sufficient. ## AI Enrichment When your Trino instance has [AI functions](https://trino.io/docs/current/functions/ai.html) configured, the plugin can automatically enrich discovered assets: - **Auto-generate descriptions** (`ai_generate_descriptions: true`) — Uses the AI connector's `ai_gen` function to produce one-sentence descriptions for tables that have no comment. - **Auto-classify tables** (`ai_classify_tables: true`) — Uses the AI connector's `ai_classify` function to assign a category label (e.g., `analytics`, `pii`, `financial`) to each table, added as a tag like `ai-category:pii`. ### AI Setup 1. Configure an AI connector in your Trino installation (e.g., `ai.properties`) 2. Set `ai_catalog` to the catalog name of that connector 3. Enable `ai_generate_descriptions` and/or `ai_classify_tables` 4. Optionally customise `ai_classify_labels` and `ai_max_enrichments` AI enrichment is best-effort - failures are logged as warnings but do not prevent normal discovery from completing.re logged as warnings but do not prevent normal discovery from completing. ## Example Configuration ```yaml host: "trino.company.com" port: 8080 user: "marmot_reader" secure: false exclude_catalogs: - "system" - "jmx" tags: - "trino" - "production" ``` ## Configuration The following configuration options are available: | Property | Type | Required | Description | |----------|------|----------|-------------| | access_token | string | false | JWT bearer token | | ai_catalog | string | false | Name of the AI connector catalog (empty = disabled) | | ai_classify_labels | []string | false | Custom classification labels | | ai_classify_tables | bool | false | Auto-classify tables into categories | | ai_generate_descriptions | bool | false | Auto-generate descriptions for undocumented tables | | ai_max_enrichments | int | false | Max tables to enrich with AI (0 = unlimited) | | catalog | string | false | Specific catalog to discover (all if empty) | | exclude_catalogs | []string | false | Catalogs to skip | | external_links | []ExternalLink | false | External links to show on all assets | | filter | Filter | false | Filter discovered assets by name (regex) | | host | string | false | Trino coordinator hostname | | include_catalogs | bool | false | Create catalog-level assets | | include_columns | bool | false | Include column info in table metadata | | include_stats | bool | false | Collect table statistics (can be slow) | | password | string | false | Password (requires HTTPS) | | port | int | false | Trino coordinator port | | secure | bool | false | Use HTTPS | | ssl_cert_path | string | false | Path to TLS certificate file | | tags | TagsConfig | false | Tags to apply to discovered assets | | user | string | false | Username for authentication | ## Available Metadata The following metadata fields are available: | Field | Type | Description | |-------|------|-------------| | catalog | string | Parent catalog name | | catalog | string | Parent catalog name | | catalog_name | string | Trino catalog name | | column_name | string | Column name | | comment | string | Table comment | | data_type | string | Column data type | | is_nullable | string | YES or NO | | ordinal_position | int | Column position | | row_count | int64 | Estimated row count | | schema | string | Parent schema name | | schema_name | string | Schema name | | table_name | string | Table or view name | | table_type | string | BASE TABLE or VIEW | --- ## Plugins Plugins automatically discover and catalog your data assets in Marmot. They connect to external systems, extract metadata and lineage, and create asset entries with minimal effort. Marmot isn't limited to plugin-based ingestion. You can also use: ### Infrastructure as Code - [Terraform Provider](/docs/Populating/Terraform) - Manage Marmot assets as Terraform resources - [Pulumi Package](/docs/Populating/Pulumi) - Integrate Marmot with Pulumi infrastructure definitions These approaches enable version-controlled asset definitions and integration with existing infrastructure workflows. ### API The [Marmot API](/docs/Populating/API) lets you programmatically create, update, and manage assets. ## Available Plugins --- ## API Marmot provides a comprehensive RESTful API to programmatically interact with your data catalog. This API allows you to create, read, update, and delete assets, establish lineage relationships, and manage metadata through simple HTTP requests. ## API Documentation [The complete Marmot API documentation can be found here.](/api) ## Authentication Authentication is required for all API requests using an API key. Add your key to all requests in the following header: ``` X-API-Key: YOUR_API_KEY ``` --- ## CLI The `ingest` command discovers metadata from configured data sources and catalogs them as assets in Marmot. It supports multiple data sources, can establish lineage relationships between assets and can attach documentation to assets. ## Installation See the [CLI Reference](/docs/cli) for configuring the host, API key and other global options. ## Configuration File The ingest command requires a YAML configuration file that defines the data sources to ingest. The configuration follows this structure: ```yaml name: my_pipeline_name runs: - source_type1: # source-specific configuration - source_type2: # source-specific configuration ``` Where `source_type` is one of the supported data source types. You can find all [available source types and their configuration in the Plugins documentation.](/docs/Plugins) Give your pipeline a unique name. This is used to track the state of the ingestion. ## Example: Ingesting Kafka Topics ```yaml runs: - kafka: bootstrap_servers: "kafka-broker:9092" client_id: "marmot-kafka-plugin" client_timeout_seconds: 60 authentication: type: "sasl_plaintext" username: "username" password: "password" mechanism: "PLAIN" schema_registry: url: "http://schema-registry:8081" enabled: true config: basic.auth.user.info: "username:password" ``` This configuration connects to a Kafka broker at `kafka-broker:9092` with SASL PLAIN authentication and integrates with a Schema Registry at `http://schema-registry:8081`. ```bash marmot ingest -c config.yaml ``` ## Where Plugins Run Discovery runs wherever the CLI runs, not on the Marmot server. The CLI connects to your data sources directly and pushes the discovered assets to the Marmot API. This means the machine running `marmot ingest` needs network access to the data sources, while the Marmot server does not: it only receives the results. On the first ingest, the CLI downloads the core plugins from `ghcr.io/marmotdata/plugins` and caches them under `~/.marmot/plugins/cache`. Later runs load them straight from the cache. Two environment variables control this: - `MARMOT_PLUGINS_AUTOINSTALL=false` disables the download, for example on air-gapped runners with pre-installed plugins - `MARMOT_PLUGINS_REGISTRY` installs from a registry mirror instead of GHCR ## Running in CI Because plugins run local to the CLI, ingestion works anywhere the CLI can run, such as a GitHub Actions workflow on a schedule: ```yaml jobs: ingest: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - name: Install Marmot CLI run: curl -fsSL get.marmotdata.io | sh - name: Cache Marmot plugins uses: actions/cache@v4 with: path: ~/.marmot/plugins/cache key: marmot-plugins-${{ runner.os }} - name: Ingest run: marmot ingest -c config.yaml env: MARMOT_HOST: https://marmot.example.com MARMOT_API_KEY: ${{ secrets.MARMOT_API_KEY }} ``` Caching `~/.marmot/plugins/cache` is optional; without it the CLI re-downloads the plugins on each run. --- ## Kubernetes Operator Ingest assets into your catalog declaratively using the Marmot Operator. Instead of running `marmot ingest` from a CLI script or the UI, the operator lets you define ingestion pipelines as Kubernetes resources. Each pipeline discovers assets from a data source and syncs them into Marmot, while the cluster handles scheduling, retries and lifecycle for you. Pipeline config lives alongside your other manifests, so changes go through the same review and GitOps workflow as everything else. The operator watches `Run` resources and reconciles them into Kubernetes Jobs or CronJobs. Each ingestion job runs as its own pod, so you can scope credentials per data source rather than giving a single Marmot instance access to all your assets. Separate pods also keep large or frequent ingestion runs from competing with each other. The operator is deployed alongside Marmot via the Helm chart. See the [Helm / Kubernetes](/docs/Deploy/Helm) guide to install Marmot first. ## Enabling the Operator Enable the operator in your Helm values: ```yaml operator: enabled: true ``` ```bash helm upgrade marmot marmotdata/marmot -f values.yaml ``` ## Creating a Run A `Run` resource defines an ingestion pipeline. The `spec.runs` array uses the same format as the [CLI configuration file](/docs/Populating/CLI#configuration-file). ```yaml apiVersion: runs.marmotdata.io/v1alpha1 kind: Run metadata: name: my-pipeline spec: schedule: "0 */6 * * *" runs: - postgresql: host: "db.example.com" port: 5432 database: "production" user: "readonly" ``` The resource's `metadata.name` is used as the pipeline name for tracking ingestion state. ```bash kubectl apply -f my-pipeline.yaml ``` ## Pod Labels and Annotations Use `podLabels` and `podAnnotations` to integrate ingestion pods with service meshes, observability tools or policy engines. On AWS, this is particularly useful for providing credentials to plugins via [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). Instead of storing AWS credentials in your pipeline config, annotate the pod so it automatically receives IAM permissions: ```yaml spec: podAnnotations: eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/marmot-s3-reader" podLabels: team: data-engineering runs: - s3: bucket: "my-data-lake" region: "eu-west-1" ``` ## Manual Triggers Trigger a scheduled pipeline outside its cron window by annotating the Run: ```bash kubectl annotate run my-pipeline runs.marmotdata.io/trigger=true ``` This creates a temporary Job that runs immediately and cleans up after 60 seconds. ## Teardown on Delete By default, deleting a Run resource runs `marmot ingest --destroy` to remove all assets that pipeline previously discovered from Marmot. Set `teardownOnDelete: false` if you want to keep existing assets after removing the Run. ## Reference ### Run Spec | Field | Type | Default | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------- | | `runs` | array | required | Source configurations, same format as CLI YAML | | `schedule` | string | | Cron expression. When set, creates a CronJob instead of a Job | | `suspend` | boolean | `false` | Pause scheduled executions. Only applies when `schedule` is set | | `concurrencyPolicy` | `Allow` / `Forbid` / `Replace` | `Forbid` | How to handle concurrent Job executions | | `backoffLimit` | int | `3` | Retries before marking a Job as failed | | `activeDeadlineSeconds` | int | | Maximum duration (seconds) a Job may run | | `successfulJobsHistoryLimit` | int | `3` | Successful CronJob runs to retain | | `failedJobsHistoryLimit` | int | `1` | Failed CronJob runs to retain | | `resources` | [ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#resourcerequirements-v1-core) | | CPU/memory requests and limits for the ingestion container | | `podLabels` | map | | Additional labels applied to the pod template | | `podAnnotations` | map | | Additional annotations applied to the pod template | | `teardownOnDelete` | boolean | `true` | Run `marmot ingest --destroy` when the Run is deleted | ### Operator Helm Values | Key | Default | Description | | ------------------------- | ---------------------- | ----------------------------------------- | | `operator.enabled` | `false` | Enable the operator Deployment and CRD | | `operator.replicas` | `1` | Number of operator replicas | | `operator.leaderElect` | `true` | Enable leader election for HA | | `operator.watchNamespace` | `""` (all) | Restrict to a single namespace | | `operator.marmot.url` | auto-detected | Marmot API URL passed to Job pods | | `operator.resources` | 100m/128Mi, 500m/256Mi | Operator pod resource requests and limits | ## Next Steps --- ## Pulumi Using Pulumi with the Marmot Terraform provider provides a powerful "Data Catalog as Code" approach, allowing you to define, version control, and automate your data catalog infrastructure, or integrate with your existing infrastructure pipelines, all with the added benefits of your preferred programming language. ## Getting Started ### Setting Up the Provider First, add the Marmot Terraform provider to your Pulumi project: ```bash $ pulumi package add terraform-provider marmotdata/marmot ``` Follow the instructions provided to link the generated SDK into your project. ### Using the Provider After adding the Marmot provider, you can use it in your Pulumi program: #### Go Example ```go package main "github.com/pulumi/pulumi-terraform-provider/sdks/go/marmot/v3/marmot" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { // Get configuration conf := config.New(ctx, "") apiKey := conf.RequireSecret("marmotApiKey") // Configure the Marmot provider provider, err := marmot.NewProvider(ctx, "marmot-provider", &marmot.ProviderArgs{ Host: pulumi.String("http://localhost:8080"), ApiKey: apiKey, }) if err != nil { return err } // Create a Marmot asset databaseAsset, err := marmot.NewAsset(ctx, "customer-database", &marmot.AssetArgs{ Name: pulumi.String("customer-database"), Type: pulumi.String("Database"), Description: pulumi.String("PostgreSQL database for customer data"), Services: pulumi.StringArray{pulumi.String("PostgreSQL")}, Tags: pulumi.StringArray{pulumi.String("database"), pulumi.String("customer")}, Metadata: pulumi.StringMap{ "owner": pulumi.String("data-team"), "version": pulumi.String("13.4"), }, }, pulumi.Provider(provider)) if err != nil { return err } // Export the asset ID and MRN ctx.Export("databaseAssetId", databaseAsset.ResourceId) ctx.Export("databaseAssetMrn", databaseAsset.Mrn) return nil }) } ``` #### TypeScript Example ```typescript const config = new pulumi.Config(); const apiKey = config.requireSecret("marmotApiKey"); // Configure the Marmot provider const provider = new marmot.Provider("marmot-provider", { host: "http://localhost:8080", apiKey: apiKey, }); // Create a Marmot asset const databaseAsset = new marmot.Asset( "customer-database", { name: "customer-database", type: "Database", description: "PostgreSQL database for customer data", services: ["PostgreSQL"], tags: ["database", "customer"], metadata: { owner: "data-team", version: "13.4", }, }, { provider }, ); // Export the asset ID and MRN export const databaseAssetId = databaseAsset.resourceId; export const databaseAssetMrn = databaseAsset.mrn; ``` ### Core Resources The Marmot provider offers these primary resources: #### `marmot.Asset` Define data assets in your catalog. Refer to the [Terraform provider documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/asset) for all available configuration options. #### `marmot.Lineage` Establish data lineage relationships between assets. Refer to the [Terraform provider documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/lineage) for all available configuration options. ## Learn More - [Pulumi Documentation](https://www.pulumi.com/docs/) - [Pulumi "Any Terraform Provider" Documentation](https://www.pulumi.com/registry/packages/terraform-provider/) - [Marmot Terraform Provider Documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs) - [Terraform Provider Repository for Marmot](https://github.com/marmotdata/terraform-provider-marmot) --- ## Terraform Using Terraform, you can manage Marmot as code, declaring your Marmot resources and automating them alongside the rest of your infrastructure. ## Getting Started ### Provider Configuration To use the Marmot Terraform provider, add it to your Terraform configuration: ```hcl terraform { required_providers { marmot = { source = "marmotdata/marmot" } } } provider "marmot" { host = "http://localhost:8080" # or the MARMOT_HOST environment variable api_key = var.marmot_api_key # or the MARMOT_API_KEY environment variable } ``` ### Authentication The provider authenticates with a Marmot API key, set through the `api_key` attribute or the `MARMOT_API_KEY` environment variable. A bearer `token` (or `MARMOT_TOKEN`) is also supported, and when no credential is provided the provider falls back to the Marmot CLI credentials from `marmot login`. To keep the secret entirely out of state, inject it using a Terraform [ephemeral resource](https://developer.hashicorp.com/terraform/language/resources/ephemeral) (Terraform >= 1.10). For example, with Google Secret Manager: ```hcl ephemeral "google_secret_manager_secret_version" "marmot_api_key" { secret = "marmot-api-key" version = "latest" } provider "marmot" { host = "https://your-marmot-host.com" api_key = ephemeral.google_secret_manager_secret_version.marmot_api_key.secret_data } ``` The same pattern works with any provider that exposes secrets as an ephemeral resource, such as AWS Secrets Manager or HashiCorp Vault. ## Resources The Marmot provider offers these primary resources: ### Assets Register the datasets, services, and other resources in your platform as assets: ```hcl resource "marmot_asset" "customer_orders" { name = "customer-orders" type = "Database" services = ["PostgreSQL"] tags = ["orders", "customer", "customer-orders"] } ``` Reference a resource's own attributes instead of hardcoding names and IDs. The asset updates in the same `terraform apply` as the resource it describes, so its metadata never drifts from what's actually deployed. #### Google Cloud Register a BigQuery table alongside its definition: ```hcl resource "google_bigquery_dataset" "analytics" { dataset_id = "analytics" location = "US" } resource "google_bigquery_table" "orders" { dataset_id = google_bigquery_dataset.analytics.dataset_id table_id = "orders" } resource "marmot_asset" "orders" { name = google_bigquery_table.orders.table_id type = "Table" services = ["BigQuery"] tags = ["orders", "analytics"] metadata = { project = google_bigquery_table.orders.project dataset = google_bigquery_dataset.analytics.dataset_id location = google_bigquery_dataset.analytics.location } } ``` #### AWS Register a DynamoDB table the same way: ```hcl resource "aws_dynamodb_table" "orders" { name = "orders" billing_mode = "PAY_PER_REQUEST" hash_key = "order_id" attribute { name = "order_id" type = "S" } } resource "marmot_asset" "orders" { name = aws_dynamodb_table.orders.name type = "Table" services = ["DynamoDB"] tags = ["orders", "analytics"] metadata = { arn = aws_dynamodb_table.orders.arn hash_key = aws_dynamodb_table.orders.hash_key billing_mode = aws_dynamodb_table.orders.billing_mode } } ``` #### Azure Register an Azure Table Storage table the same way: ```hcl resource "azurerm_storage_account" "analytics" { name = "analyticsdata" resource_group_name = "analytics" location = "East US" account_tier = "Standard" account_replication_type = "LRS" } resource "azurerm_storage_table" "orders" { name = "orders" storage_account_name = azurerm_storage_account.analytics.name } resource "marmot_asset" "orders" { name = azurerm_storage_table.orders.name type = "Table" services = ["Azure Table Storage"] tags = ["orders", "analytics"] metadata = { storage_account = azurerm_storage_account.analytics.name location = azurerm_storage_account.analytics.location } } ``` See the [`marmot_asset` documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/asset) for all available configuration options. ### Lineage Describe how data flows between assets to build a lineage graph: ```hcl resource "marmot_asset" "order_processor" { name = "order-processor" type = "Service" services = ["Kubernetes"] } resource "marmot_lineage" "orders_to_processor" { source = marmot_asset.customer_orders.mrn target = marmot_asset.order_processor.mrn } ``` See the [`marmot_lineage` documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/lineage) for all available configuration options. ### Glossary Terms Define shared business terminology and organize it hierarchically: ```hcl resource "marmot_glossary_term" "active_customer" { name = "Active Customer" definition = "A customer with at least one order in the last 90 days." metadata = { domain = "sales" } } ``` See the [`marmot_glossary_term` documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/glossary_term) for all available configuration options. ### Teams Manage the teams that own catalog entities: ```hcl resource "marmot_team" "analytics" { name = "analytics" description = "Owns the reporting datasets" } ``` See the [`marmot_team` documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/team) for all available configuration options. ### Users Manage user accounts. The password is set through the write-only `password_wo` attribute (Terraform >= 1.11), so it never lands in state: ```hcl ephemeral "random_password" "alice" { length = 24 } resource "marmot_user" "alice" { name = "Alice Nguyen" username = "alice" password_wo = ephemeral.random_password.alice.result password_wo_version = "1" role_names = ["admin"] } ``` Change `password_wo_version` to push a new password on a later apply. See the [`marmot_user` documentation](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/user) for all available configuration options. ### Data Products Group related assets into a data product. Teams and users own it through `owner_team_ids` and `owner_user_ids`: ```hcl resource "marmot_data_product" "orders" { name = "orders" description = "Order events and the tables derived from them" tags = ["orders"] owner_team_ids = [marmot_team.analytics.id] } ``` Assets join a data product directly with `marmot_data_product_asset`: ```hcl resource "marmot_data_product_asset" "orders_customer" { data_product_id = marmot_data_product.orders.id asset_id = marmot_asset.customer_orders.id } ``` Or dynamically with `marmot_data_product_rule`, which matches assets by a search query or a metadata pattern: ```hcl resource "marmot_data_product_rule" "order_datasets" { data_product_id = marmot_data_product.orders.id name = "order-datasets" type = "query" query_expression = "tag:orders" } ``` See the [`marmot_data_product`](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/data_product), [`marmot_data_product_asset`](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/data_product_asset), and [`marmot_data_product_rule`](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs/resources/data_product_rule) documentation for all available configuration options. ## Learn More - Full documentation for [the Marmot provider on the Terraform Registry](https://registry.terraform.io/providers/marmotdata/marmot/0.4.0/docs) - A [full example](https://github.com/marmotdata/terraform-provider-marmot/tree/main/examples/full) in the provider repository --- ## UI Create and manage ingestion pipelines directly from the Marmot web interface. ## Managing Pipelines The **Runs** page displays all pipelines with their status, schedule, and last run time. You can run, edit, or delete pipelines, and view run history in the **Run History** tab. ## Creating a Pipeline Navigate to **Runs** and click **Create Pipeline** to open the pipeline wizard. ### Step 1: Basic Info Enter a unique name for your pipeline, e.g. `daily-postgres-sync`. ### Step 2: Choose Plugin Select the data source you want to discover assets from. Use the search box to quickly find the plugin you need. ### Step 3: Configure Configure the connection settings for your chosen plugin. Options vary by data source but typically include host, authentication, and discovery settings. ### Step 4: Schedule Set a CRON schedule for automated runs, or leave as manual to run on-demand. --- ## Populating Your Catalog There are many ways to populate Marmot with assets and lineage. Use all of these methods together, or just the ones that fit your existing workflows. ## Methods --- ## SDK The Marmot SDK is a typed client for the REST API, available in **Python**, **Go** and **TypeScript**. Authentication resolves automatically from environment variables, cached OAuth tokens or workload identity. ## Install ```bash pip install marmot-sdk ``` Requires Python 3.10+. Package name is `marmot-sdk`, import name is `marmot`. ```bash go get github.com/marmotdata/marmot/sdk/go ``` Requires Go 1.24+. Import path is `github.com/marmotdata/marmot/sdk/go`, package name is `marmot`. ```bash pnpm add @marmotdata/sdk ``` Requires Node 18+. ESM and CJS builds ship together with bundled types. ## Authenticate Every SDK resolves credentials from the same priority chain, so the same code runs locally, in CI and in production without branching on environment: 1. **Explicit arguments.** `api_key` / `token` passed to `connect()` or `NewClient()`. 2. **Environment variables.** `MARMOT_API_KEY`, `MARMOT_TOKEN`, `MARMOT_HOST`, `MARMOT_CONTEXT`. 3. **Cached OAuth token.** Written to `~/.config/marmot/credentials.json` by `marmot login`. 4. **Workload identity.** GitHub Actions OIDC, GCP metadata or a Kubernetes service-account token. No API key needed. If no credential resolves, the SDK raises an `AuthError` so misconfiguration fails fast. Log in for local use: ```bash marmot login http://localhost:5173 ``` Then construct a client: ```python from marmot import AuthenticatedApiClient, UsersApi # Resolves host and credential from the chain client = AuthenticatedApiClient.connect() # Or supply them explicitly client = AuthenticatedApiClient.connect( host="https://marmot.example.com", api_key="..." ) me = UsersApi(client).get_users_me_sync() print(me.name, "via", client.credential.source) ``` ```go package main "context" "log" "os" marmot "github.com/marmotdata/marmot/sdk/go" ) func main() { ctx := context.Background() // Resolves from the chain client, err := marmot.NewClient(marmot.ClientOptions{}) if err != nil { log.Fatal(err) } // Or pass credentials explicitly client, err = marmot.NewClient(marmot.ClientOptions{ Host: "https://marmot.example.com", APIKey: os.Getenv("MARMOT_API_KEY"), }) } ``` ```ts // Resolves from the chain const client = await connect(); // Or pass an API key explicitly const explicit = await connect({ baseUrl: "https://marmot.example.com", apiKey: "...", }); ``` The following sections all assume `client` (and `ctx` for Go) is already constructed as shown above. **Python:** one `AuthenticatedApiClient` is shared by every generated `*Api` class, which you construct around it — `UsersApi(client)`, `AssetsApi(client)`, and so on. Method names follow the operation: `get_assets_id`, `post_lineage_batch`. Each exists twice, as a coroutine and with a `_sync` suffix that runs it on a shared event loop, so the snippets below stay synchronous. Request bodies are pydantic models from `marmot.generated.models`, and failures raise `marmot.errors` types (`NotFoundError`, `AuthError`, `ValidationError`, `RateLimitError`, `ServerError`) rather than returning a status code. ## Search One unified search across assets, glossary terms, teams and data products. Returns a typed `SearchResponse` with facets, results and pagination. ```python from marmot import AuthenticatedApiClient, SearchApi client = AuthenticatedApiClient.connect() results = SearchApi(client).get_search_sync(q="orders", types=["Table", "Topic"], limit=20) print(f"{results.total} matches") for hit in results.results or []: print(hit.name, "-", hit.type.value if hit.type else "unknown") ``` ```go "fmt" marmot "github.com/marmotdata/marmot/sdk/go" ) results, err := client.Search.Query(ctx, "orders", marmot.SearchOptions{ Types: []string{"Table", "Topic"}, Limit: 20, }) if err != nil { return err } for _, hit := range results.Results { fmt.Println(hit.Name, hit.Metadata["mrn"]) } ``` ```ts const client = await connect(); const results = await client.search("orders", { types: ["Table", "Topic"], limit: 20, }); for (const hit of results.results ?? []) { console.log(hit.name, hit.metadata?.mrn); } ``` Marmot accepts both free-text queries and a structured query language (`@type: "Table" AND @provider: "postgres"`). See the [Query Language guide](/docs/queries) for the full grammar. ## Assets Every catalog entry is an Asset. The Assets resource covers CRUD, lookup by natural key, search, summary aggregates and tag management. ### Fetch by ID ```python from marmot import AssetsApi, AuthenticatedApiClient client = AuthenticatedApiClient.connect() asset = AssetsApi(client).get_assets_id_sync(id="01HX...") print(asset.name, asset.mrn) ``` ```go asset, err := client.Assets.Get(ctx, "01HX...") if err != nil { return err } fmt.Println(asset.Name, asset.Mrn) ``` ```ts const client = await connect(); const asset = await client.assets.get("01HX..."); console.log(asset.name, asset.mrn); ``` ### Lookup by natural key When you know an asset by `(type, service, name)` but not its ID, `lookup` resolves it. `find` does the same but returns `nil` / `None` instead of raising on a miss. ```python from marmot import AssetsApi, AuthenticatedApiClient from marmot.errors import NotFoundError assets = AssetsApi(AuthenticatedApiClient.connect()) asset = assets.get_assets_lookup_type_service_name_sync( type="Table", service="postgres", name="orders" ) # A missing asset raises rather than returning None try: assets.get_assets_lookup_type_service_name_sync( type="Table", service="postgres", name="nope" ) except NotFoundError: asset = None ``` ```go asset, err := client.Assets.Lookup(ctx, marmot.LookupInput{ Type: "Table", Service: "postgres", Name: "orders", }) if err != nil { return err } // nil, nil on 404 instead of an error maybe, err := client.Assets.Find(ctx, marmot.LookupInput{ Type: "Table", Service: "postgres", Name: "orders", }) ``` ```ts const client = await connect(); const asset = await client.assets.lookup({ type: "Table", service: "postgres", name: "orders", }); // null on 404 instead of throwing const maybe = await client.assets.find({ type: "Table", service: "postgres", name: "orders", }); ``` ### Search and summary ```python from marmot import AssetsApi, AuthenticatedApiClient assets = AssetsApi(AuthenticatedApiClient.connect()) hits = assets.get_assets_search_sync( q="customer", types=["Table"], services=["postgres"], tags=["pii"], limit=50, ) summary = assets.get_assets_summary_sync() # totals by type, provider, tag ``` ```go hits, err := client.Assets.Search(ctx, marmot.AssetSearchOptions{ Query: "customer", Types: []string{"Table"}, Providers: []string{"postgres"}, Tags: []string{"pii"}, Limit: 50, }) summary, err := client.Assets.Summary(ctx) ``` ```ts const client = await connect(); const hits = await client.assets.search({ query: "customer", types: ["Table"], providers: ["postgres"], tags: ["pii"], limit: 50, }); const summary = await client.assets.summary(); ``` ### Create, update, delete ```python from marmot import AssetsApi, AuthenticatedApiClient from marmot.generated.models import CreateAssetRequest, UpdateAssetRequest assets = AssetsApi(AuthenticatedApiClient.connect()) created = assets.post_assets_sync( create_asset_request=CreateAssetRequest( name="orders", type="Table", providers=["postgres"], metadata={"owner": "data-eng"}, ) ) updated = assets.put_assets_id_sync( id=created.id, update_asset_request=UpdateAssetRequest(description="Customer orders"), ) assets.delete_assets_id_sync(id=created.id) ``` ```go created, err := client.Assets.Create(ctx, marmot.CreateAssetInput{ Name: "orders", Type: "Table", Providers: []string{"postgres"}, Tags: []string{"pii"}, }) if err != nil { return err } _, err = client.Assets.Update(ctx, *created.ID, marmot.UpdateAssetInput{ Description: "Customer orders", }) err = client.Assets.Delete(ctx, *created.ID) ``` ```ts const client = await connect(); const created = await client.assets.create({ name: "orders", type: "Table", providers: ["postgres"], metadata: { owner: "data-eng" }, }); const updated = await client.assets.update(created.id!, { description: "Customer orders", }); await client.assets.delete(created.id!); ``` ### Tag management ```python from marmot import AssetsApi, AuthenticatedApiClient from marmot.generated.models import TagRequest assets = AssetsApi(AuthenticatedApiClient.connect()) assets.post_assets_tags_id_sync(id=asset_id, tag_request=TagRequest(tag="pii")) assets.delete_assets_tags_id_sync(id=asset_id, tag_request=TagRequest(tag="pii")) ``` ```go err := client.Assets.AddTag(ctx, assetID, "pii") if err != nil { return err } err = client.Assets.RemoveTag(ctx, assetID, "pii") ``` ```ts const client = await connect(); await client.assets.addTag(assetId, "pii"); await client.assets.removeTag(assetId, "pii"); ``` ## Lineage Lineage edges identify endpoints by MRN (`:///`). Read the graph from any node; write one edge or many at a time. ### Read the graph ```python from marmot import AuthenticatedApiClient, LineageApi lineage = LineageApi(AuthenticatedApiClient.connect()) graph = lineage.get_lineage_assets_id_sync(id=asset_id, direction="both", limit=50) upstream = lineage.get_lineage_assets_id_sync(id=asset_id, direction="upstream", limit=10) # Leave out edge types you don't want, e.g. structural CONTAINS edges flow = lineage.get_lineage_assets_id_sync(id=asset_id, exclude_types="CONTAINS") ``` ```go graph, err := client.Lineage.Get(ctx, assetID, marmot.LineageOptions{ Direction: "both", Limit: 50, }) if err != nil { return err } upstream, err := client.Lineage.Upstream(ctx, assetID, marmot.LineageOptions{Limit: 10}) ``` ```ts const client = await connect(); const graph = await client.lineage.get(assetId, { direction: "both", depth: 3, }); const upstream = await client.lineage.upstream(assetId, { depth: 2 }); const downstream = await client.lineage.downstream(assetId, { depth: 2 }); ``` ### Write edges Prefer `/lineage/direct` and `/lineage/batch` for new integrations. They accept simple `(source, target)` pairs and de-duplicate server-side. ```python from marmot import AuthenticatedApiClient, LineageApi from marmot.generated.models import LineageEdge lineage = LineageApi(AuthenticatedApiClient.connect()) # Single edge lineage.post_lineage_direct_sync( lineage_edge=LineageEdge( source="postgres://prod/sales/orders", target="kafka://prod/orders.events", ) ) # Batched: one HTTP call, many edges lineage.post_lineage_batch_sync( lineage_edge=[ LineageEdge( source="postgres://prod/sales/orders", target="kafka://prod/orders.events", ), LineageEdge( source="kafka://prod/orders.events", target="s3://prod/orders-archive", ), ] ) ``` ```go // Single edge _, err := client.Lineage.Write(ctx, marmot.WriteEdgeInput{ Source: "postgres://prod/sales/orders", Target: "kafka://prod/orders.events", }) if err != nil { return err } // Batched: one HTTP call, many edges _, err = client.Lineage.Batch(ctx, []marmot.WriteEdgeInput{ {Source: "postgres://prod/sales/orders", Target: "kafka://prod/orders.events"}, {Source: "kafka://prod/orders.events", Target: "s3://prod/orders-archive"}, }) ``` ```ts const client = await connect(); await client.lineage.write({ source: "postgres://prod/sales/orders", target: "kafka://prod/orders.events", }); await client.lineage.batch([ ["postgres://prod/sales/orders", "kafka://prod/orders.events"], ["kafka://prod/orders.events", "s3://prod/orders-archive"], ]); ``` Leave `Type` empty (`DIRECT` is the default) for code-derived edges; set it explicitly (`"writes"`, `"AGENT_LOOKUP"`, …) when you want to distinguish causes in the lineage graph. ## Glossary Business glossary terms with definitions, descriptions and hierarchies via `parent_term_id`. ```python from marmot import AuthenticatedApiClient, GlossaryApi from marmot.generated.models import CreateTermRequest, UpdateTermRequest glossary = GlossaryApi(AuthenticatedApiClient.connect()) page = glossary.get_glossary_list_sync(limit=50) print(f"{len(page.terms or [])} of {page.total} terms") hits = glossary.get_glossary_search_sync(q="customer") term = glossary.post_glossary_sync( create_term_request=CreateTermRequest( name="PII", definition="Personally Identifiable Information", description="Data that can identify an individual.", ) ) glossary.put_glossary_id_sync( id=term.id, update_term_request=UpdateTermRequest(name="Personally Identifiable Information"), ) glossary.delete_glossary_id_sync(id=term.id) ``` ```go page, err := client.Glossary.List(ctx, marmot.GlossaryListOptions{Limit: 50}) if err != nil { return err } hits, err := client.Glossary.Search(ctx, marmot.GlossarySearchOptions{Query: "customer"}) term, err := client.Glossary.Create(ctx, marmot.CreateTermInput{ Name: "PII", Definition: "Personally Identifiable Information", Description: "Data that can identify an individual.", }) _, err = client.Glossary.Update(ctx, *term.ID, marmot.UpdateTermInput{ Name: "Personally Identifiable Information", }) err = client.Glossary.Delete(ctx, *term.ID) ``` ```ts const client = await connect(); const page = await client.glossary.list({ limit: 50 }); const hits = await client.glossary.search({ query: "customer" }); const term = await client.glossary.create({ name: "PII", definition: "Personally Identifiable Information", description: "Data that can identify an individual.", }); await client.glossary.update(term.id!, { name: "Personally Identifiable Information", }); await client.glossary.delete(term.id!); ``` ## Users & Teams ```python from marmot import AuthenticatedApiClient, TeamsApi, UsersApi client = AuthenticatedApiClient.connect() users, teams = UsersApi(client), TeamsApi(client) me = users.get_users_me_sync() user = users.get_users_id_sync(id=user_id) page = users.get_users_sync(active=True, limit=100) all_teams = teams.get_teams_sync() team = teams.get_teams_id_sync(id=team_id) members = teams.get_teams_id_members_sync(id=team_id) ``` ```go me, err := client.Users.Me(ctx) if err != nil { return err } user, err := client.Users.Get(ctx, userID) active := true users, err := client.Users.List(ctx, marmot.UsersListOptions{Active: &active, Limit: 100}) teams, err := client.Teams.List(ctx, marmot.TeamsListOptions{}) team, err := client.Teams.Get(ctx, teamID) members, err := client.Teams.Members(ctx, teamID) ``` ```ts const client = await connect(); const me = await client.users.me(); const user = await client.users.get(userId); const users = await client.users.list({ active: true, limit: 100 }); const teams = await client.teams.list(); const team = await client.teams.get(teamId); const members = await client.teams.members(teamId); ``` ## API Keys Manage personal API keys for the authenticated user. The full key token is only readable from the `create` response, so store it immediately. ```python from marmot import AuthenticatedApiClient, UsersApi from marmot.generated.models import CreateAPIKeyRequest users = UsersApi(AuthenticatedApiClient.connect()) keys = users.get_users_apikeys_sync() created = users.post_users_apikeys_sync( create_api_key_request=CreateAPIKeyRequest(name="ci-deploy", expires_in_days=30) ) print(created.key) # only readable here users.delete_users_apikeys_id_sync(id=created.id) ``` ```go "fmt" marmot "github.com/marmotdata/marmot/sdk/go" ) keys, err := client.APIKeys.List(ctx) if err != nil { return err } created, err := client.APIKeys.Create(ctx, marmot.CreateAPIKeyInput{ Name: "ci-deploy", ExpiresInDays: 30, }) fmt.Println(created.Key) // only readable here err = client.APIKeys.Delete(ctx, *created.ID) ``` ```ts const client = await connect(); const keys = await client.apiKeys.list(); const created = await client.apiKeys.create({ name: "ci-deploy", expiresInDays: 30, }); console.log(created.key); // only readable here await client.apiKeys.delete(created.id!); ``` ## Runs Read pipeline-ingestion run history. Useful when wiring up alerts on failed ingests or audit dashboards. ```python from marmot import AuthenticatedApiClient, RunsApi runs = RunsApi(AuthenticatedApiClient.connect()) recent = runs.get_runs_sync(statuses="failed,running", limit=20) run = runs.get_runs_id_sync(id=run_id) entities = runs.get_runs_id_entities_sync(id=run_id, status="failed") ``` ```go recent, err := client.Runs.List(ctx, marmot.RunsListOptions{ Statuses: "failed,running", Limit: 20, }) if err != nil { return err } run, err := client.Runs.Get(ctx, runID) entities, err := client.Runs.Entities(ctx, runID, marmot.RunEntitiesOptions{ Status: "failed", }) ``` ```ts const client = await connect(); const recent = await client.runs.list({ statuses: "failed,running", limit: 20, }); const run = await client.runs.get(runId); const entities = await client.runs.entities(runId, { status: "failed" }); ``` ## Metrics Catalog usage and breakdown metrics. `top_assets` and `top_queries` take an inclusive `[start, end]` window of RFC3339 timestamps. ```python from marmot import AuthenticatedApiClient, MetricsApi metrics = MetricsApi(AuthenticatedApiClient.connect()) total = metrics.get_metrics_assets_total_sync() print(total.count) by_type = metrics.get_metrics_assets_by_type_sync() by_provider = metrics.get_metrics_assets_by_provider_sync() top = metrics.get_metrics_top_assets_sync( start="2025-01-01T00:00:00Z", end="2025-02-01T00:00:00Z", limit=10, ) queries = metrics.get_metrics_top_queries_sync( start="2025-01-01T00:00:00Z", end="2025-02-01T00:00:00Z", limit=10, ) ``` ```go total, err := client.Metrics.TotalAssets(ctx) if err != nil { return err } byType, err := client.Metrics.AssetsByType(ctx) byProvider, err := client.Metrics.AssetsByProvider(ctx) top, err := client.Metrics.TopAssets(ctx, marmot.TopOptions{ Start: "2025-01-01T00:00:00Z", End: "2025-02-01T00:00:00Z", Limit: 10, }) queries, err := client.Metrics.TopQueries(ctx, marmot.TopOptions{ Start: "2025-01-01T00:00:00Z", End: "2025-02-01T00:00:00Z", Limit: 10, }) ``` ```ts const client = await connect(); const total = await client.metrics.totalAssets(); const byType = await client.metrics.assetsByType(); const byProvider = await client.metrics.assetsByProvider(); const top = await client.metrics.topAssets({ start: "2025-01-01T00:00:00Z", end: "2025-02-01T00:00:00Z", limit: 10, }); const queries = await client.metrics.topQueries({ start: "2025-01-01T00:00:00Z", end: "2025-02-01T00:00:00Z", limit: 10, }); ``` ## Owners Search the catalog for asset owners (users and teams). ```python from marmot import AuthenticatedApiClient, OwnersApi owners = OwnersApi(AuthenticatedApiClient.connect()) hits = owners.get_owners_search_sync(q="alice", limit=10) for owner in hits.owners or []: print(owner) ``` ```go hits, err := client.Owners.Search(ctx, "alice", marmot.OwnerSearchOptions{Limit: 10}) ``` ```ts const client = await connect(); const hits = await client.owners.search("alice", { limit: 10 }); ``` ## Admin Trigger or poll a full search reindex. Requires admin permissions. ```python from marmot import AdminApi, AuthenticatedApiClient admin = AdminApi(AuthenticatedApiClient.connect()) accepted = admin.post_admin_search_reindex_sync() status = admin.get_admin_search_reindex_sync() print(status.running, status.es_configured) ``` ```go _, err := client.Admin.Reindex(ctx) if err != nil { return err } status, err := client.Admin.ReindexStatus(ctx) fmt.Println(status.Running, status.EsConfigured) ``` ```ts const client = await connect(); const accepted = await client.admin.reindex(); const status = await client.admin.reindexStatus(); console.log(status.running, status.es_configured); ``` --- ## API Reference Marmot exposes a REST API for managing assets, lineage, glossary terms, data products, and more. View the full API documentation → --- ## Asset Rules Asset Rules automatically apply enrichments to assets matching specific criteria. Define a rule once and Marmot keeps everything in sync as your Catalog changes, including new assets that match. ## Creating a Rule Navigate to **Asset Rules** under **Governance** in the header and click **Create Rule**. The creation flow has three steps. ### Basic Information Give your rule a unique name and an optional description. ### Enrichments Choose what to apply to matching assets. A rule must include at least one of: - **External links** — runbooks, dashboards, wiki pages or monitoring URLs. Each link has a name, URL and optional icon. - **Glossary terms** — select one or more terms from your existing glossary to associate with matching assets. ### Query Define which assets the rule should match using Marmot's query language (the same syntax used in search). For example: - `@type: "table" AND @provider: "postgres"` — all PostgreSQL tables - `@tag: "pii"` — any asset tagged as PII - `@metadata.owner = "platform-team"` — assets owned by a specific team Use the **Preview** button to see which assets currently match before saving. ## How Rules Are Applied Rules are evaluated every 30 minutes by default and whenever a new asset is added to the Catalog. Only rules whose configuration or matching assets have changed are re-evaluated. When multiple rules match the same asset, all enrichments are applied. Rules can be enabled or disabled at any time. Disabled rules retain their configuration so you can re-enable them later. ## Managing Rules The Asset Rules page lists all rules with their match count, number of links and terms, enabled status and last updated time. Click any rule to view its configuration or see matched assets. Rules can be edited, enabled, disabled or deleted from the detail page. Changes take effect on the next reconciliation cycle, or you can trigger an immediate evaluation by updating the rule. --- ## CLI Reference The Marmot CLI lets you interact with your data catalog directly from the terminal. ## Installation --- ## Authentication Sign in once with `marmot login`. It opens your browser, you sign in the way you always do, and the CLI keeps a token for you. ```bash marmot login https://marmot.example.com ``` That is all most people need. Every other command uses the token from then on. It lasts 24 hours; run `marmot login` again when it has expired. ### Useful options ```bash # Skip the browser and print the sign-in link instead. # Open it anywhere, then paste the address the browser ends up on back here. marmot login https://marmot.example.com --no-launch-browser # Sign in again even if you still have a valid token marmot login https://marmot.example.com --force # Print the token, for scripts marmot login https://marmot.example.com --print-token # Forget the token marmot logout ``` ### What happens behind the scenes The CLI opens the Marmot sign-in page in your browser and waits on a local port for the browser to come back with a code. It swaps that code for a token and saves it in `~/.config/marmot/credentials.json`. The token is a signed JWT with your user id, your roles and their permissions, and an expiry. This is plain OAuth 2.0 with PKCE. There is no client secret and nothing to configure. ### Pushing to a registry on your Marmot host Login also makes the token available to `docker`, `crane` and `oras` for the Marmot host, so those tools can push to a registry served there without a `docker login`. This needs `docker-credential-marmot` on your `PATH`. It is just the marmot binary under another name, and the install script sets it up. If you installed marmot some other way: ```bash ln -s "$(command -v marmot)" "$(dirname "$(command -v marmot)")/docker-credential-marmot" ``` Without it, login writes the token straight into `~/.docker/config.json`. That works until the token expires, and not at all if Docker Desktop manages your credentials. Login tells you when that is the case. ### API Keys API keys can still be used and always take priority over cached login tokens. ```bash # Via flag marmot assets list --api-key mrmot_abc123 # Via environment variable export MARMOT_API_KEY=mrmot_abc123 ``` ### Auth Priority When multiple credentials are available, the CLI uses the first one found: 1. `--api-key` flag or `MARMOT_API_KEY` environment variable 2. Cached OAuth token from `marmot login` (for the active context) 3. Kubernetes service account token (auto-detected in-cluster) --- ## Contexts Contexts let you work with multiple Marmot instances (e.g. staging and production). A context is created automatically when you run `marmot login`. ```bash # Login creates a context named after the hostname marmot login https://marmot.example.com # → Context "marmot.example.com" created and activated. marmot login https://staging.marmot.dev # → Context "staging.marmot.dev" created and activated. # List all contexts (* = active) marmot context list # marmot.example.com https://marmot.example.com (token valid) # * staging.marmot.dev https://staging.marmot.dev (token valid) # Switch active context marmot context use marmot.example.com # Remove a context and its cached token marmot context delete staging.marmot.dev ``` --- ## Configuration You can also configure the CLI with flags, environment variables or a config file. These are checked in order of precedence. ### CLI Flags ```bash marmot assets list --host https://marmot.example.com --api-key my-key ``` ### Environment Variables ```bash export MARMOT_HOST=https://marmot.example.com export MARMOT_API_KEY=my-key ``` ### Config File ```bash marmot config init ``` This creates `~/.config/marmot/config.yaml` interactively. You can also use `marmot config set ` to set individual values. | Key | Description | Default | | --- | --- | --- | | `host` | Marmot server URL | `http://localhost:8080` | | `api_key` | API key for authentication | (none) | | `output` | Default output format (`table`, `json`, `yaml`) | `table` | | `current_context` | Active context name | (none) | --- ## Output Formats All commands support `--output` / `-o` with `table` (default), `json` or `yaml`. ```bash marmot assets list -o json | jq '.assets[].name' ``` --- ## Commands All list commands support `--limit` and `--offset` for pagination. Destructive commands prompt for confirmation unless `--yes` is passed. Run `marmot --help` for full flag details. ### marmot login ``` marmot login [url] [flags] ``` Authenticate with a Marmot instance via browser using OAuth 2.0 PKCE. A valid cached token is reused. If no URL is provided and no context is active, prompts for one. Creates a context automatically and registers the token with the Docker credential store for the instance's host. | Flag | Description | | --- | --- | | `--force` | Sign in again even if a valid token is cached | | `--print-token` | Print the access token on stdout; status messages go to stderr | | `--no-launch-browser` | Print the sign-in URL instead of opening a browser; the callback URL can be pasted on stdin | ### marmot logout ``` marmot logout ``` Remove the cached authentication token for the active context, and the registry credential login registered for its host. ### marmot context ``` marmot context ``` Manage named contexts for switching between Marmot instances. Contexts are created automatically by `marmot login`. | Subcommand | Description | | --- | --- | | `list` | Show all contexts with token status | | `use ` | Switch active context | | `delete ` | Remove context and its cached token | ### marmot assets ``` marmot assets [flags] ``` Browse, search and manage assets in your catalog. Use `list` and `search` with `--types`, `--providers` and `--tags` to filter results. ### marmot search ``` marmot search [flags] ``` Unified search across assets, glossary terms, teams and users. Filter by result type with `--types`. ### marmot glossary ``` marmot glossary [flags] ``` Manage glossary terms. Create terms with `--name` and `--definition`, optionally nesting them under a parent with `--parent-id`. ### marmot runs ``` marmot runs [flags] ``` View pipeline ingestion runs. Filter with `--pipelines` and `--statuses`. ### marmot lineage ``` marmot lineage get [flags] ``` View the upstream and downstream lineage graph for an asset. Control traversal depth with `--depth`. ### marmot users ``` marmot users [flags] ``` View user information. `me` shows the currently authenticated user. ### marmot apikeys ``` marmot apikeys [flags] ``` Manage API keys for authentication. The full key is only shown once at creation time. ### marmot teams ``` marmot teams [flags] ``` View teams and their members. ### marmot metrics ``` marmot metrics

[flags] ``` View catalog metrics and usage statistics. `top-assets` and `top-queries` require a time range via `--start` and `--end` (RFC3339 format, defaults to the last 30 days). ### marmot admin ``` marmot admin ``` Administrative operations. `reindex` triggers a full search reindex and `reindex-status` checks its progress. ### marmot config ``` marmot config ``` Manage CLI configuration. See [Configuration](#configuration) above for details. --- ## Tab Completion Generate shell completions with `marmot completion `. Supported shells are `bash`, `zsh`, `fish` and `powershell`. ```bash source <(marmot completion bash) ``` --- ## Next Steps --- ## Data Products Data Products let you group related assets into logical collections. A "Customer Analytics" product might bundle together a PostgreSQL table storing profiles, a Kafka topic with real-time events, an API endpoint and a dashboard. Instead of navigating hundreds of individual assets, teams can discover and understand related data as a cohesive unit. ## Creating a Data Product Navigate to **Data Products** in the header and click **Create Product**. Give your product a name and description, optionally add tags for categorisation, and assign owners responsible for the product. ## Adding Assets There are two ways to populate a Data Product with assets. **Manual assignment** lets you add specific assets directly. Open the product, go to the **Assets** and search for what you want to include. This works well when you have a known set of assets that belong together. **Dynamic rules** use Marmot's query language to automatically include assets matching certain criteria. Rules continuously evaluate as your catalog changes, so new assets matching the criteria are added automatically. To add a rule, go to the **Rules** tab, click **Add Rule** and enter a name along with the query. For example, `@metadata.owner = "analytics-team"` would include all assets owned by that team, while `@type: "topic" AND @provider: "kafka"` would include all Kafka topics. --- ## Glossary The Glossary lets you define business terms and create a shared vocabulary across your organisation. Instead of different teams using different names for the same concept, the glossary establishes standard terminology that everyone can reference. ## Creating Terms Navigate to **Glossary** in the header and click **Create Term**. Give your term a name and description explaining what it means and how it should be used. ## Associating Terms with Assets Glossary terms become useful when linked to data assets. To associate a term with an asset: 1. Navigate to the asset page 2. Find **Glossary Terms** in the sidebar 3. Click **Add** and search for the term 4. Select the term to link it --- ## Introduction Marmot is the open source **context layer** for your whole stack: a single catalog for every asset your systems and teams depend on, from services, APIs, queues, topics and brokers to databases, tables and pipelines. It exists to solve **context starvation**, the moment an engineer or an AI agent has to act without knowing what exists, who owns it, what it means, or what it connects to. Marmot is built so both **humans and agents** can ask that question and get a real answer. [Catalog your assets](Populating/index.md) once, enrich them with ownership and business context, and expose them through the UI, a [REST API](api-reference.md), and a built-in [MCP server](MCP/index.md) that lets AI agents read your catalog, then write back the [lineage](open-lineage.md) they generate. ## Built for agents AI agents are only as good as the context they can reach. Through a built-in MCP server and our SDKs, Marmot gives your agents a live, governed map of every asset in your stack: what exists, who owns it, what it means, and how it all connects. ## Why Marmot? Most catalogs were built to help a data team document tables. Marmot is built to feed context to whoever needs it, human or agent, across every kind of asset: services, APIs, queues, topics, brokers, databases, tables and pipelines. That means agents are first class, not an afterthought. A native MCP server and our SDKs are part of the core, so your agents read the same governed context your team does. And Marmot stays light enough to actually adopt: a single binary backed only by PostgreSQL, with no platform team required. ## Architecture Marmot is built entirely in Go with PostgreSQL being the only external dependency, handling search, job scheduling and metadata storage. Unlike traditional catalogs that have opinionated ingestion methods, Marmot lets you populate your catalog however you like. ## What Marmot stores Marmot is a context layer, so it stores **metadata about your assets**, not the data inside them. That means schemas, field names and types, ownership, descriptions, tags, lineage and statistics. The rows in your tables, the messages on your topics and the payloads behind your APIs never enter Marmot. Plugins read a source's structure and metadata into PostgreSQL; the data itself never moves. The easiest path is to run it on our platform, isolated per customer and under strict access controls. Need everything to stay within your control? Run Marmot yourself, free or with an enterprise license, in your own cloud on AWS, Google Cloud, Azure, OVHcloud or anywhere else. ## Features Everything you need to turn scattered assets into a context layer that humans and agents can both query. ## Get started Pick a starting point. The [Quick Start](quick-start.md) walks you from an empty deployment to a populated catalog, step by step. --- ## Metrics ## Overview Marmot collects various application metrics for both Prometheus monitoring and built-in dashboards in the UI. ## Prometheus/OpenMetrics Endpoints You can enable metrics in the configuration to expose a Prometheus/OpenMetrics endpoint on `/metrics`. This endpoint does not have auth enabled, you should configure Prometheus to scrape the endpoints for each Marmot instance you have deployed. **values.yaml:** ```yaml metrics: enabled: true port: 9090 ``` **Environment variables:** ```bash MARMOT_METRICS_ENABLED=true MARMOT_METRICS_PORT=9090 ``` ## Helm Chart The Helm chart creates a ServiceMonitor for Prometheus Operator: ```bash helm install marmot ./chart --set config.metrics.enabled=true --set monitoring.serviceMonitor.enabled=true ``` ```yaml config: metrics: enabled: true port: 9090 monitoring: serviceMonitor: enabled: true interval: 30s ``` ## Endpoints - `/metrics` - Prometheus endpoint (no auth) - `/api/v1/metrics` - UI dashboard API (requires auth) --- ## OpenLineage [OpenLineage](https://openlineage.io/) is an open standard for data lineage collection and analysis. It provides a unified way to track data flows across different tools and platforms by emitting standardised events during job execution. Marmot integrates with OpenLineage to automatically discover assets and lineage relationships from your data pipelines, eliminating manual catalog maintenance. :::note[Compatibility] OpenLineage support in Marmot is still experimental and has not been tested with all sources. Please report any issues you encounter on GitHub. ::: ## What You Get ## Supported Asset Types Marmot maps OpenLineage events to specific asset types: | Asset Type | Description | | ---------- | ------------------------ | | `DAG` | Airflow workflows | | `Task` | Individual Airflow tasks | | `Model` | DBT models | | `Project` | DBT projects | | `Table` | Database tables | | `File` | Data files | | `Topic` | Kafka topics | ## Authentication By default, the OpenLineage endpoint requires authentication via an API key. You can disable authentication for trusted environments if needed. ### Generate API Key 1. Navigate to **Profile** → **API Keys** 2. Click **New Key** 3. Copy the generated key 4. Configure your OpenLineage producer ### Endpoint URL ``` POST /api/v1/lineage Authorization: X-API-Key ``` ### Disable Authentication To disable authentication for the OpenLineage endpoint, set the following configuration: **Config file** ```yaml openlineage: auth: enabled: false ``` **Environment variable:** ```bash export MARMOT_OPENLINEAGE_AUTH_ENABLED=false ``` :::warning Disabling authentication allows anyone to send lineage events to your Marmot instance. Only use this in trusted environments. ::: ## Configuration Examples ### Airflow Configure the OpenLineage provider in `airflow.cfg`: ```ini [openlineage] transport = http url = https://your-marmot-instance.com/api/v1/lineage api_key = your-api-key ``` ### DBT Add to your `profiles.yml`: ```yaml your_profile: outputs: prod: # ... your connection details vars: openlineage: url: https://your-marmot-instance.com/api/v1/lineage api_key: your-api-key ``` ### Spark Set environment variables: ```bash export OPENLINEAGE_URL=https://your-marmot-instance.com/api/v1/lineage export OPENLINEAGE_API_KEY=your-api-key ``` --- ## Query Language Marmot provides a query language for searching and filtering assets in your catalog. The language supports free-text search, field-specific filters, comparison operators and boolean logic. :::tip Optional but Powerful The query language is entirely optional. Simple free-text searches work well for everyday discovery. When you need precision, such as finding all Kafka topics owned by a specific team or tables with more than a million rows, the query language gives you that control. Queries are also repeatable and shareable, making it easy to bookmark common searches or share them with your team. ::: ## Where It's Used The query language powers several features across Marmot: ## Query Builder The search bar includes a visual query builder that helps you construct queries without memorising the syntax. Click the filter icon to open it, select your field and operator, then enter your value. The builder generates the query syntax automatically. ## Syntax Reference ### Fields Filter assets using field prefixes: | Field | Description | Example | | ----- | ----------- | ------- | | `@type` | Asset type | `@type: "table"` | | `@provider` | Provider or platform | `@provider: "kafka"` | | `@name` | Asset name | `@name: "users"` | | `@kind` | Resource kind in Marmot | `@kind: "asset"` | | `@metadata.*` | Custom metadata fields | `@metadata.team: "platform"` | Metadata supports dot notation for nested fields: `@metadata.config.retention: "7d"` ### Operators | Operator | Description | Example | | -------- | ----------- | ------- | | `:` or `=` | Exact match | `@type: "table"` | | `!=` | Not equal | `@metadata.environment != "test"` | | `contains` | Substring match | `@name contains "customer"` | | `>` `<` `>=` `<=` | Numeric comparison | `@metadata.partitions > 10` | | `range` | Numeric range | `@metadata.size range [100 TO 500]` | | `*` | Wildcard | `@name: "customer*"` | ### Boolean Logic Combine filters with `AND`, `OR` and `NOT`. Use parentheses to control precedence: ```marmot # Multiple conditions @type: "topic" AND @provider: "kafka" # Either condition @metadata.priority: "high" OR @metadata.criticality: "critical" # Exclusion @metadata.environment: "production" AND NOT @name contains "test" # Grouped logic (@type: "table" OR @type: "view") AND @provider: "postgres" ``` ## Examples Search across asset names, descriptions and metadata without any special syntax. ```marmot user orders ``` Find all Kafka topics by combining type and provider filters. ```marmot @type: "topic" AND @provider: "kafka" ``` Find all assets owned by a specific team using custom metadata. ```marmot @metadata.team: "data-platform" ``` Filter assets based on numeric metadata values. ```marmot @type: "topic" AND @metadata.partitions > 10 ``` Use wildcards when you don't know the exact name. ```marmot @name: "*customer*" AND @type: "table" ``` Use parentheses to control how conditions are combined. ```marmot (@type: "table" OR @type: "view") AND @provider: "postgres" ``` --- ## Quick Start Get Marmot running in seconds with Docker Compose. [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed. Create a `docker-compose.yaml`: ```yaml services: marmot: image: ghcr.io/marmotdata/marmot:latest ports: - "8080:8080" environment: MARMOT_DATABASE_HOST: postgres MARMOT_DATABASE_PORT: 5432 MARMOT_DATABASE_USER: marmot MARMOT_DATABASE_PASSWORD: marmot MARMOT_DATABASE_NAME: marmot MARMOT_DATABASE_SSLMODE: disable MARMOT_SERVER_ALLOW_UNENCRYPTED: true depends_on: postgres: condition: service_healthy postgres: image: postgres:16-alpine environment: POSTGRES_USER: marmot POSTGRES_PASSWORD: marmot POSTGRES_DB: marmot volumes: - marmot_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U marmot"] interval: 5s timeout: 5s retries: 5 volumes: marmot_data: ``` ```bash docker compose up -d ``` Open [http://localhost:8080](http://localhost:8080) and log in with `admin` / `admin`. ## Next Steps --- ## What Is an AI Context Layer? A 2026 Guide [← All resources](/resources) An AI context layer is the governed source of truth that tells AI agents what your data means before they act. It is the difference between an agent that guesses about your schemas and ownership, and one that asks, gets a real answer scoped to its permissions, and acts on facts. This guide explains what an AI context layer is, why agents need one, how it differs from RAG, and how to build one on a data catalog with MCP. ## What is an AI context layer? {#what-is} **An AI context layer is the layer that gives AI agents real, governed context about your data at the moment they need it.** It exposes structured metadata, schemas, descriptions, ownership, lineage, glossary terms and quality signals, over an interface an agent can call, usually a plain API and [MCP](/resources/mcp-for-data). When an agent is about to write a query, answer a question or trigger an action, it queries the context layer first. It asks what exists, what a field means, who owns it and what it connects to, and gets an answer limited to what its credentials are allowed to see. At its core the layer holds the meaning and governance around your data, which is exactly what an agent is missing on its own. The richer context layers go further and broker governed access to the underlying data itself, so an agent can move from understanding your data to working with it through the same permission model. ## Metadata context and data context {#metadata-and-data} **An AI context layer comes in two scopes, and the most capable ones cover both.** The first is metadata context: the structure, meaning, ownership, lineage and governance of your data. This is what an agent needs to understand a landscape and decide what to use. The second is data context: governed access to the actual data in the underlying sources, so an agent can read real values rather than only descriptions of them, through the same permissions and policies. A request like "show me last week's failed orders" needs both, metadata to know which table to use and what the fields mean, and data access to return the rows. The strongest context layers govern both kinds of access behind one interface, so an agent moves from understanding your data to working with it without ever stepping outside what it is allowed to see. ## Why AI agents need a context layer {#why} **The model is rarely the limit. The context is.** A capable model still has no idea that your `orders` table excludes refunds, that `customer_id` joins to a different system, or that a dashboard is owned by a team that deprecated it last quarter. Left to guess, it produces a confident, plausible, wrong answer. A context layer closes that gap. It gives the agent the same facts a careful engineer would check first: the schema, the definition, the owner, the lineage and whether the data can be trusted. With that in hand the agent stops hallucinating about your stack and starts acting on what is actually there. As agents take on more real work, this is the part that decides whether they are useful or dangerous. We go deeper on what agents actually need in [AI data engineering](/resources/ai-data-engineering). ## What an AI context layer provides {#what-it-provides} A useful context layer serves more than a list of table names. It provides: - **Structure.** Schemas and field-level detail, so an agent knows the shape of the data. - **Meaning.** Descriptions and a business glossary, so "active customer" resolves to one definition. - **Ownership.** Who is responsible for an asset, so an agent can attribute and escalate correctly. - **Lineage.** What feeds what, so the agent understands dependencies and impact. - **Trust signals.** Freshness, certification and quality, so the agent knows what to rely on. - **Governance.** Every answer scoped to the caller's permissions, so an agent sees only what it should. ## How an AI context layer works {#how-it-works} **The context layer sits between your data estate and your AI tools, and answers questions at runtime.** Metadata flows in from across the stack, databases, warehouses, queues, pipelines and dashboards, and is exposed to agents through two interfaces: - A **plain API**, for programmatic access from any tool or script. - **[MCP](/resources/mcp-for-data)**, the Model Context Protocol, so assistants like Claude and Cursor discover and call the layer's tools in natural language without bespoke wiring. The critical detail is that access is governed at the point of the query. The agent authenticates, usually with an API key, and every response is scoped to that key's permissions. It never gets a raw dump of the whole estate, only the context it is allowed to see. That is what makes a context layer safe to put in front of an autonomous agent. ## AI context layer vs RAG {#vs-rag} **RAG and a context layer solve different problems, and the best systems use both.** Retrieval-augmented generation finds relevant passages in unstructured documents and pastes them into the prompt. It is excellent for "what does this policy say" or "summarise these tickets". An AI context layer is about the structure and governance of your data, not free text. It tells the agent what tables exist, what their columns mean, who owns them and how they connect, on demand and scoped to permissions. RAG answers from documents; the context layer tells the agent what the underlying data actually is. An agent answering a data question well often needs both: RAG for the prose, the context layer for the facts about the data itself. ## Building an AI context layer {#building} **The most practical way to build an AI context layer is on a data catalog with native MCP.** A [data catalog](/resources/data-catalog) already collects the metadata an agent needs, schemas, ownership, lineage, glossary and quality, from across your stack. Adding a governed, agent-queryable interface turns that inventory into a context layer. When you evaluate options, weigh the things that decide whether the layer is actually usable: - **Native MCP support**, so agents connect without a separate integration per tool. - **Governed context**, with every query scoped to the caller rather than handing back the whole catalog. - **Coverage**, so the layer can see enough of your stack to be worth querying, through connectors, SDKs and infrastructure-as-code. - **A small footprint**, so you can stand the layer up and keep it current without operating a platform. - **Sane cost under agent load**, because agents query far more often than people, and per-usage pricing adds up fast. We compare the main catalogs on exactly these terms in [Data Catalogs as the AI Context Layer](/resources/data-catalogs-for-ai-agents). ## Frequently asked questions {#faq} ### What is an AI context layer? {#faq-what} An AI context layer is the governed source of truth that tells AI agents what your data means before they act. It exposes structured metadata, schemas, ownership, lineage, glossary terms and quality signals, over an interface like an API or MCP, so an agent can ask what exists, what it means and who owns it, and get an answer scoped to its permissions. In practice it is usually built on a data catalog. ### Why do AI agents need a context layer? {#faq-why} An AI agent is only as good as the context it is given. Without one it guesses about your schemas, ownership and meaning, and a confident wrong answer is worse than no answer. A context layer gives the agent real, governed information about your data at runtime, so it stops hallucinating about your stack and acts on facts instead. The model is rarely the limit; the context usually is. ### Does an AI context layer serve data or just metadata? {#faq-data-or-metadata} Both, depending on the layer. At minimum it serves metadata context: the structure, meaning, ownership, lineage and governance of your data. The most capable context layers also serve data context, brokering governed access to the actual data in the underlying sources, so an agent can read real values rather than only descriptions of them. A request like "show me last week's failed orders" needs both: metadata to find the right table and data access to return the rows. Either way, access stays scoped to the caller's permissions. ### What is the difference between an AI context layer and RAG? {#faq-vs-rag} RAG retrieves unstructured documents and pastes relevant passages into the prompt. An AI context layer serves structured, governed metadata about your data estate, schemas, ownership, lineage and definitions, that an agent queries on demand. RAG is about text; a context layer is about the shape and governance of your data. They are complementary: RAG answers from documents, the context layer tells the agent what the underlying data actually is. ### Is an AI context layer the same as a data catalog? {#faq-vs-catalog} Not quite. A data catalog is the inventory of metadata. An AI context layer is what that catalog becomes when it exposes its metadata to agents in a governed, queryable way, usually over MCP. A modern data catalog with native MCP is the most practical way to build an AI context layer, because it already holds the schemas, ownership and lineage an agent needs. ### How do you build an AI context layer? {#faq-build} Build it on a data catalog that collects metadata from across your stack and exposes it to agents over an API and MCP, with access scoped to each caller. Populate it with schemas, ownership, lineage, glossary terms and quality signals, then connect your AI tools to it. The lighter the catalog is to run and the more native its MCP support, the faster you have a context layer in production. ### What role does MCP play in an AI context layer? {#faq-mcp} MCP, the Model Context Protocol, is the standard interface that lets AI tools reach a context layer. The catalog runs an MCP server that exposes tools such as search, find ownership and traverse lineage, and assistants like Claude and Cursor call them in natural language. MCP is how the context layer becomes available to agents without a bespoke integration per tool. ## Related {#related} - [Data Catalogs as the AI Context Layer: A 2026 Comparison](/resources/data-catalogs-for-ai-agents) - [What Is a Data Catalog?](/resources/data-catalog) - [MCP for Data: Connecting AI Agents to Your Catalog](/resources/mcp-for-data) - [AI Data Engineering](/resources/ai-data-engineering) --- ## AI Data Engineering: Giving Agents Real Context [← All resources](/resources) AI data engineering is the work of giving LLMs and agents real, governed context about your data so they can build and reason without guessing. It is the part of working with AI that decides whether an agent is genuinely useful or just confidently wrong. This guide explains what AI data engineering is, why context matters more than the model, what context an agent actually needs, and how to deliver it at runtime. ## What is AI data engineering? {#what-is} **AI data engineering is the discipline of preparing data and its context so AI systems can use it safely and well.** The term covers two related ideas. One is using AI to help do data engineering: generating pipelines, writing transformations, explaining models. The other, and the harder one, is engineering the context that AI tools need to work against your data at all. This guide focuses on the second. A model is good at reasoning in general. It knows nothing about your `orders` table, your ownership structure or what "active customer" means in your organisation. AI data engineering is the work of making that knowledge available, structured, current and governed, so an agent can use it. ## Why context matters more than the model {#why} **The model is rarely the limit. The context is.** An assistant with no context about your landscape will hallucinate table names, invent schemas and guess at ownership, because it has nothing real to anchor on. Swapping in a larger model does not fix this. Giving it a catalog with real schemas, lineage and owners does, and the output changes completely. This is why teams that get value from agents tend to invest less in prompt tricks and more in the context behind the prompt. We wrote about the failure mode in [AI is making assumptions about your data](/blog/ai-is-making-assumptions-about-your-data). The fix is not a cleverer model, it is real context delivered at the moment the agent needs it. ## What context an AI agent needs {#what-context} An agent needs the same things a careful engineer would check before touching unfamiliar data: - **Schemas.** What tables and fields exist, and their types. - **Ownership.** Who is responsible, so the agent routes questions and attributes correctly. - **Lineage.** What feeds what, so it understands dependencies and impact. - **Glossary.** What business terms actually mean in your organisation. - **Quality signals.** Freshness and certification, so it prefers trusted sources over deprecated ones. All of it scoped to the agent's permissions, so it sees only what it is allowed to. ## AI data engineering vs traditional data engineering {#vs-traditional} **Traditional data engineering serves humans. AI data engineering adds the agent as a consumer.** Classic data engineering moves and shapes data into pipelines, models and dashboards that people read. That work does not go away. What is new is a consumer that needs the meaning and governance around the data, not just the data, and needs it at runtime through an interface it can query. A dashboard is rendered for a person to interpret. An agent has to fetch structured context and act on it directly. So the added job is making context machine-readable, governed and queryable, which is what the rest of this guide is about. ## How agents get that context {#how} **The practical path in 2026 is a catalog that exposes context over [MCP](/resources/mcp-for-data) and a clean API.** The agent retrieves governed context at runtime, scoped to its permissions, rather than working from a stale README or a prompt someone pasted in last quarter. That runtime delivery matters. Context pasted into a prompt is out of date the moment a schema changes. Context pulled from a catalog over MCP reflects the current state of your stack every time the agent asks. This governed, on-demand delivery is what turns a pile of metadata into a usable [AI context layer](/resources/ai-context-layer). ## How to do AI data engineering well {#best-practices} A few habits carry most of the value: - **Catalog the whole stack**, not just the warehouse, so an agent sees the real landscape. - **Keep context current** by ingesting from the systems of record, not hand-maintained docs. - **Make it queryable at runtime** over MCP and an API, rather than baking it into prompts. - **Govern every query**, scoping context to the caller so agents stay in their lane. - **Carry quality and ownership** alongside schemas, so the agent can judge what to trust. ## Frequently asked questions {#faq} ### What is AI data engineering? {#faq-what} AI data engineering is the work of giving LLMs and AI agents real, governed context about your data so they can build and reason without guessing. It covers two things: using AI to help do data engineering, and engineering the context those AI tools need, the schemas, ownership, lineage and definitions an agent must have to act correctly. The second is the harder and higher-leverage part. ### Why is context more important than the model for AI agents? {#faq-context-vs-model} A capable model with no context about your landscape will hallucinate table names, invent schemas and guess at ownership, because it has nothing real to work from. The limit is rarely the model's reasoning, it is what the model knows about your specific data. Give it real schemas, lineage and owners and the output changes completely. Context is the lever, not a bigger model. ### What context does an AI agent need about data? {#faq-what-context} An agent needs schemas, so it knows what tables and fields exist and their types; ownership, so it routes questions and attributes correctly; lineage, so it understands what feeds what and what a change will break; a glossary, so business terms resolve to one meaning; and quality signals, so it can tell a trusted asset from a deprecated one. All of it scoped to the agent's permissions. ### How is AI data engineering different from traditional data engineering? {#faq-vs-traditional} Traditional data engineering moves and shapes data for human consumers: pipelines, models and dashboards. AI data engineering adds a new consumer, the agent, which needs the meaning and governance around the data, not just the data itself, and it needs it at runtime through an interface it can query. The pipelines still matter; the new work is making the context machine-readable and governed. ### How do you give an LLM context about your data? {#faq-how} The practical path in 2026 is a data catalog that exposes context over MCP and a clean API. The catalog holds schemas, ownership, lineage and glossary from across your stack, and the agent retrieves what it needs at runtime, scoped to its permissions, rather than working from a stale README or a hand-pasted prompt. That keeps the context current and governed. ### What is a data context layer? {#faq-context-layer} A data context layer is the governed source an agent queries to understand your data before it acts. It exposes structured metadata, and increasingly the underlying data itself, through an interface like MCP or an API, scoped to the caller. AI data engineering is largely the work of building and maintaining that layer so agents work from facts rather than guesses. ## Related {#related} - [What Is an AI Context Layer?](/resources/ai-context-layer) - [MCP for Data: Connecting AI Agents to Your Catalog](/resources/mcp-for-data) - [What Is a Data Catalog?](/resources/data-catalog) - [Data Catalogs as the AI Context Layer: A 2026 Comparison](/resources/data-catalogs-for-ai-agents) --- ## What Is a Data Catalog? A 2026 Guide [← All resources](/resources) A data catalog is the inventory of your data: what exists, who owns it, how it connects and what it means. It pulls metadata from across your stack into one place you can search and trust. For years that was a tool for people. In 2026 the more important consumer is often the AI agent that queries the catalog at runtime before it writes code or answers a question. This guide explains what a data catalog is, what it does, the features and benefits that matter, the main types, and how to choose one now that agents are a first-class consumer. ## What is a data catalog? {#what-is-a-data-catalog} **A data catalog is an organised inventory of an organisation's data assets, built from the metadata that describes them.** It collects metadata from databases, warehouses, object storage, message queues, pipelines and dashboards into one searchable place, and records what each asset is, who owns it, what it means and how it relates to everything else. The data itself stays where it lives. The catalog holds the context around it: schemas, descriptions, owners, tags, glossary terms, quality signals and lineage. That context is what turns a sprawl of disconnected systems into something a person, or an agent, can navigate with confidence. ## What does a data catalog do? {#what-it-does} A good data catalog does four things, and most of its value comes from doing all four together. - **Inventory.** A single searchable list of tables, topics, buckets, models and dashboards across every system, so nobody has to remember where things live. - **Context.** Schemas, descriptions, owners, tags and business glossary terms, so an asset means the same thing to everyone who uses it. - **Lineage.** A map of how data flows, so you can see what feeds a dashboard and what breaks if you change a table upstream. - **Governance.** Ownership and access controls, so the right people, and the right agents, see the right things and nothing they should not. ## Why data catalogs matter in 2026 {#why-it-matters} **The catalog used to be a productivity tool for humans. Now it is the context layer for AI.** When an agent writes a query, answers a question or triggers an action, it needs to know what your data means before it acts. Without that context it guesses, and a confident wrong answer is worse than no answer. A data catalog is where that context lives, and the way agents reach it has standardised around two interfaces: a plain API and, increasingly, [MCP](/resources/mcp-for-data), the Model Context Protocol. An agent asks the catalog what exists, what it means and who owns it, gets an answer scoped to its permissions, and only then acts. This is the shift that has reshaped what a catalog is for, and it is covered in depth in [AI data engineering](/resources/ai-data-engineering). The volume changes too. An agent queries metadata far more often than a person does, so how the catalog exposes context, and what it costs to serve those queries, now matters as much as how it looks in a UI. ## Data catalog features to look for {#features} Whether you are evaluating tools or building a shortlist, these are the features that separate a useful catalog from a glorified spreadsheet. - **Search and discovery.** Fast, fuzzy search across every asset, with filters by source, owner, tag and domain. - **Rich metadata and a business glossary.** Schemas, descriptions and shared definitions, so "active customer" means one thing. - **Column and table lineage.** Upstream and downstream flow, for impact analysis and root-cause work. - **Ownership and access control.** Clear owners and permissions, so governance is enforced rather than documented. - **Data quality signals.** Freshness, certification and test results, so users know what to trust. - **An agent-queryable interface.** A clean API and native [MCP](/resources/mcp-for-data) support, so AI tools like Claude and Cursor reach governed context, not raw dumps. - **Broad integrations.** Connectors, SDKs and infrastructure-as-code paths to populate the catalog from the stack you already run. - **A sensible footprint.** What you have to deploy and maintain to keep the catalog running, which ranges from a single binary to a multi-service platform. ## Types of data catalog {#types} Catalogs differ along a few axes that matter more than marketing categories. - **Open source vs commercial.** Open source catalogs such as Marmot, OpenMetadata and DataHub let you self-host, inspect the code and avoid per-seat fees. Commercial platforms such as Atlan, Collibra and Secoda are managed services with vendor support and compliance certifications. - **Self-hosted vs fully managed.** Self-hosting gives you control and keeps metadata in your own infrastructure. A managed SaaS removes the operational work in exchange for a commercial relationship and less control over where data sits. - **Lightweight vs platform.** Some catalogs run as a single process on a database. Others are multi-service platforms with their own search cluster and ingestion framework. Both can hold large catalogs; the difference is how much you have to operate. ## Benefits of a data catalog {#benefits} **A data catalog pays off by turning scattered, untrusted data into something people and agents can use quickly and safely.** The concrete benefits: - **Faster discovery.** People stop pinging colleagues to ask which table to use, and agents stop guessing. - **Trust.** Ownership, definitions and quality signals tell users whether a dataset can be relied on. - **Safer change.** Lineage shows what depends on what, so you can change a pipeline without quietly breaking a report. - **Governance and compliance.** Access is scoped and auditable, which matters more, not less, once agents can act on data. - **Agent enablement.** A governed context layer is what lets AI tools work against your data without making things up. ## How to choose a data catalog in 2026 {#how-to-choose} The criteria changed once agents became a consumer. Beyond search and lineage, weigh these: - **How it exposes context to AI.** Native [MCP](/resources/mcp-for-data) support, an agent-queryable API and lineage reachable through that interface. - **Governed context, not raw access.** Every query scoped to the caller's permissions, so an agent sees only what it should. - **Deployment footprint.** What you actually have to run and keep healthy. - **Connector coverage and integration paths.** Pre-built connectors, plus SDKs and infrastructure-as-code for the long tail. - **Cost model.** Per-seat or per-usage pricing behaves differently under agent workloads, where query volume is high. We compare the main options on exactly these terms in [Data Catalogs as the AI Context Layer](/resources/data-catalogs-for-ai-agents), with head-to-head pages for [Marmot vs DataHub](/resources/marmot-vs-datahub), [Marmot vs OpenMetadata](/resources/marmot-vs-openmetadata) and [Marmot vs Atlan](/resources/marmot-vs-atlan). ## Frequently asked questions {#faq} ### What is a data catalog? {#faq-what} A data catalog is an organised inventory of an organisation's data assets. It collects metadata from across the stack, databases, warehouses, queues, pipelines and dashboards, into one searchable place that records what data exists, who owns it, what it means and how it connects. In 2026 the catalog is also the context layer that AI agents query at runtime before they act. ### What is a data catalog used for? {#faq-used-for} A data catalog is used to find data, understand it and trust it. People use it to search for the right table or dashboard, see who owns it, read its definition and trace its lineage. AI agents use the same catalog over an API or MCP to get governed context about your data before they write a query or answer a question, so they stop guessing about schemas and ownership. ### What is the difference between a data catalog and a database? {#faq-vs-database} A database stores the data itself. A data catalog stores metadata about that data: where it lives, what the columns mean, who owns it, how fresh it is and how it flows through the stack. The catalog does not hold your records. It is the index and context layer that makes the data across all your systems discoverable and trustworthy. ### What is the best open source data catalog in 2026? {#faq-best-open-source} The leading open source data catalogs are Marmot, OpenMetadata and DataHub. Marmot is the lightest to run, a single Go binary on Postgres with a built-in MCP server for AI agents. OpenMetadata and DataHub offer the widest connector libraries but run as multi-service platforms. The right choice depends on how much infrastructure you want to run and how you plan to expose context to agents. See [the full comparison](/resources/data-catalogs-for-ai-agents). ### Do I need a data catalog? {#faq-need} If people or AI agents regularly ask which table to use, who owns a dataset or whether data can be trusted, a data catalog pays for itself. It becomes close to essential once you run AI agents against your data, because an agent needs governed context to act safely and a catalog is where that context lives. ### Is a data catalog the same as metadata management? {#faq-metadata-management} They are closely related but not identical. Metadata management is the broader practice of collecting and maintaining metadata. A data catalog is the product that puts that metadata to work: a searchable inventory with lineage, ownership, glossary and governance that people and agents actually use. Most modern catalogs are how teams do metadata management in practice. ## Related {#related} - [Data Catalogs as the AI Context Layer: A 2026 Comparison](/resources/data-catalogs-for-ai-agents) - [MCP for Data: Connecting AI Agents to Your Catalog](/resources/mcp-for-data) - [AI Data Engineering](/resources/ai-data-engineering) - [Data Governance](/resources/data-governance) --- ## Best Data Catalogs for AI Agents in 2026: The AI Context Layer Compared [← All resources](/resources) The job of a data catalog changed in 2026. The consumer is no longer just a human browsing a UI, it is an agent querying metadata at runtime. This guide compares the main data catalogs on exactly that axis: how each one exposes context to AI. We cover Marmot, OpenMetadata, DataHub, Atlan, Collibra, Secoda, Amundsen and Apache Atlas. By 2026 almost all of these tools can talk to an agent, so native MCP is no longer what separates them. The real differences are quieter: how much infrastructure you have to run to keep the context flowing, whether it is governed at the point an agent queries it, and how much of your stack the catalog can actually see. --- ## Why "context layer" is the new frame {#why-context-layer} **The catalog is now infrastructure for AI, not just a directory for people.** When agents are the consumer, metadata stops being documentation you read and becomes context an agent retrieves to avoid guessing. The catalog's value is measured by how cleanly it can serve that context to a model. We made the underlying argument in [AI is making assumptions about your data, and getting them wrong](/blog/ai-is-making-assumptions-about-your-data). An LLM with no context about your data landscape will hallucinate table names, invent schemas and guess at ownership. It produces output that looks plausible and costs you more time to correct than to write yourself. The model is not the limit. The context is. Gartner analyst Andres Garcia-Rodeja put a number on the risk in 2026, predicting that 60% of agentic analytics projects relying solely on connectivity will fail by 2028 for lack of a consistent semantic layer underneath. MCP solves the connectivity problem, how an agent reaches your data. The context layer solves the meaning problem, what your data actually is. Connectivity without context just lets the agent be wrong faster. --- ## What makes a catalog an AI context layer? {#evaluation-criteria} **A catalog earns the name when an agent can retrieve governed context from it at runtime through a standard interface, without a human in the loop.** Here are the criteria we use to judge that, and they are the columns in the matrix below. - **Native MCP support.** Does the catalog ship a Model Context Protocol server, so AI assistants can call it with no glue code? Native means built into the product. Official means the vendor ships a separate server. Community (unofficial) means a third-party server exists but is not maintained by the project. None means you build the integration yourself. - **Agent-queryable API.** Is there a clean, documented REST or GraphQL API an agent or automation can hit directly? MCP usually wraps this, so a good API is the foundation. - **Command line interface.** Is there a first-party CLI for searching, updating and scripting against the catalog, or only an ingestion or admin utility? A full CLI makes the catalog easy to wire into pipelines, CI and developer workflows, and gives agents another governed way in. - **Lineage exposed to AI.** Can an agent traverse upstream and downstream dependencies through the interface, not just see them rendered in a UI? Lineage is the highest-value context for "what breaks if I change this". - **Governed context, not raw access.** Does the interface enforce ownership, certification and access controls at query time, so the agent gets trusted context scoped to its permissions rather than a raw dump? This is what stops an agent confidently citing a deprecated table. - **Deployment footprint.** What do you have to run and keep alive? A single binary is a different operational reality from Kafka, a graph database and a search cluster. - **Connector coverage.** How much of your real stack can it actually see? Context is only as complete as the metadata it ingests. Defining the criteria up front matters because an AI context layer is only useful if it is complete, current and trusted. A catalog that nails MCP but only sees a third of your stack still leaves the agent guessing about the rest. --- ## Comparison matrix {#comparison-matrix} **Here is how the eight catalogs compare as AI context layers, as of June 2026.** MCP cells reflect what we could confirm from each vendor's current documentation and repositories. Where a vendor is actively shipping MCP, we say so; we have not assumed any tool "can't" just to flatter the column. | Tool | MCP support | CLI | Agent-queryable API | Lineage exposed to AI | Core dependencies | Deploy footprint | Connectors | Open source | Best for | |------|-------------|-----|---------------------|------------------------|-------------------|------------------|------------|-------------|----------| | **[Marmot](https://github.com/marmotdata/marmot)** | Native (built in) | Full | Yes, REST | Yes, via MCP, CLI, SDK and API | Postgres only (Elasticsearch optional) | Single Go binary | ~28 plugins + IaC | Yes (MIT) | Native MCP with the smallest footprint | | **[OpenMetadata](https://open-metadata.org)** | Native (built in) | Partial (ingestion) | Yes, REST | Yes | Elasticsearch or OpenSearch, ingestion framework | Multi-service | 120+ | Yes (Apache 2.0) | Broad coverage, open source | | **[DataHub](https://datahub.com)** | Official (separate server) | Full | Yes, REST and GraphQL | Yes, via MCP and API | Kafka, graph store, Elasticsearch | Heavy, multi-service | Extensive | Yes (Apache 2.0) | Broad ecosystem, existing Kafka stacks | | **[Atlan](https://atlan.com)** | Native (hosted) | Partial (contracts) | Yes, REST | Yes | SaaS (managed) | Hosted, none to run | Large managed library | No | Enterprise hosted context layer | | **[Collibra](https://www.collibra.com)** | Official (server) | Partial | Yes, REST | Yes | SaaS (managed) | Hosted, none to run | Extensive enterprise | No | Regulated, governance-heavy orgs | | **[Secoda](https://www.secoda.co)** | Native (hosted) | None | Yes, REST | Yes | SaaS (managed) | Hosted, none to run | Broad managed | No | AI-first hosted catalog | | **[Amundsen](https://www.amundsen.io)** | Community (unofficial) | None | Yes, REST | Yes (in UI), limited via API | Neo4j or Atlas, Elasticsearch | Multi-service | Community-driven | Yes (Apache 2.0) | Search-led discovery, existing users | | **[Apache Atlas](https://atlas.apache.org)** | Community (unofficial) | Partial (admin/import) | Yes, REST | Yes | JanusGraph, HBase, Solr, Kafka | Heavy, Hadoop-era | Hadoop ecosystem | Yes (Apache 2.0) | Hadoop and Cloudera estates | --- ## The tools, one by one {#per-tool} Each entry below follows the same shape: who it is best for, a short take, pros and cons, and when to choose it. Read the matrix for the overview, read these for the detail. ### Marmot {#marmot} **Best for:** teams that want native MCP context across their whole stack with the smallest operational footprint. Marmot is the open source catalog built for the AI context job from the ground up, and it does it with less to run than anything else here. A single Go binary on Postgres becomes an AI context layer the moment it starts, because the MCP server is part of the binary rather than a separate service to deploy. **Marmot pros:** - **Native MCP, built in.** The MCP server ships in the binary. Nothing extra to deploy, run or keep alive. - **Smallest footprint here.** One Go binary, Postgres only. No Kafka, no graph database, no required search cluster. Elasticsearch is optional, not a dependency. - **Governed by default.** Every agent query runs with the permissions of the API key behind it, so the agent gets role-scoped context, never a raw dump. - **Vendor-neutral coverage.** Catalogs Postgres, Kafka, S3, BigQuery, dbt, Airflow and more in one place, so an agent sees the whole landscape rather than one vendor's slice. - **Catalog as code.** Official Terraform and Pulumi providers (`marmot_asset`, `marmot_lineage`) let you populate assets and lineage straight from the pipelines you already run. - **Three focused MCP tools:** `discover_data` (natural language and qualified-identifier lookups, with lineage traversal and suggested next actions), `find_ownership` and `lookup_term`. - **CLI and packaged agent Skill.** A full-featured `marmot` CLI for search, lineage and glossary, plus a ready-made agent Skill so assistants can drive the catalog over the CLI, REST API or MCP without bespoke wiring. - **The widest set of integration paths here.** Plugins driven by YAML ingestion through the CLI, a Kubernetes-native operator, Terraform and Pulumi providers, fully featured Go, TypeScript and Python SDKs, a REST API and MCP. There is almost always a first-party way to get data in or out, in the language or workflow your team already uses. - **MIT licensed.** **Marmot cons:** - **Smaller connector library today.** Around 28 plugins against 120+ for OpenMetadata, in a fast-growing ecosystem. For anything without a plugin yet, the Terraform and Pulumi providers or the Go, TypeScript and Python SDKs populate assets and lineage straight from your existing infrastructure and code, so the gap is bridged rather than left open. If a source has no plugin yet, you populate it from the Terraform you are writing anyway, so the catalog grows with your existing infrastructure rather than waiting on a connector. We cover why Postgres alone is enough to back a catalog in [Postgres: One Database to Rule Them All](/blog/postgres-one-database-to-rule-them-all), and the low-infrastructure goal in [Data catalog without the complex infrastructure](/blog/data-catalog-without-complex-infrastructure). **Choose Marmot if:** you want the best open source AI context layer you can actually run, native MCP and governed context over your whole stack, stood up in minutes with no platform team to keep it alive. ### OpenMetadata {#openmetadata} **Best for:** teams that want the broadest open source coverage and have rebuilt around AI context. OpenMetadata is one of the most complete open source catalogs and has moved hard into the AI context space, branding itself an open source context layer. **OpenMetadata pros:** - **Native MCP, built into the platform.** As of June 2026 MCP is a first-class service category; clients can read and write the configured integrations, assuming the platform's roles and policies. - **Widest open source connector library**, well past 120 sources. - **Mature knowledge graph and lineage.** - **Apache 2.0 licensed.** **OpenMetadata cons:** - **Operational weight.** Expects a search backend (Elasticsearch or OpenSearch) and an ingestion framework, so you run several moving parts, not one binary. - **Heavier to stand up and maintain** than a single-process catalog. ### DataHub {#datahub} **Best for:** teams already invested in Kafka and a search stack who want a broad integration ecosystem. DataHub has a broad ecosystem and an official MCP server, published as a separate package by Acryl, with real production use behind it. Block wired its open source Goose agent to DataHub's MCP server to cut metadata lookups from hours to minutes. **DataHub pros:** - **Official MCP server.** Agents can search assets, traverse lineage, inspect schemas and generate SQL through Cursor, Claude Desktop, Windsurf and others. - **Mature lineage tooling**, including column-level lineage built up over years. - **Extensive ecosystem and integrations**, plus a GraphQL API. - **Apache 2.0 licensed.** **DataHub cons:** - **Heavy architecture.** A full deployment leans on Kafka, a graph store and Elasticsearch, more to run than a single-process catalog of the same size. - **MCP server is a separate component**, not built into the core, so it is one more thing to run and version. ### Atlan {#atlan} **Best for:** enterprises that want a fully managed, hosted context layer. Atlan has been one of the loudest voices defining the "context layer for AI agents" frame, and its product backs it. **Atlan pros:** - **Hosted, native MCP.** Connects Claude, Cursor, ChatGPT, Gemini and automation platforms like LangChain and n8n in real time. - **Read and write context:** search, lineage, metadata updates, classification and glossary management. - **Broad managed connector library**, nothing for you to deploy. **Atlan cons:** - **Closed source** and enterprise-priced. - **Query economics need scrutiny.** Per-seat or per-usage costs add up fast when an agent issues far more queries than a person. ### Collibra {#collibra} **Best for:** regulated organisations where governance is the point. Collibra approaches AI context from the governance side, and in May 2026 launched an AI Command Center as a governance control plane for agents that call tools and trigger actions. **Collibra pros:** - **MCP server** (`chip`, available in the Databricks Marketplace) exposing governed metadata, glossary queries and asset details, with more than 100 customers reported using it. - **Governance-first.** Built for auditability and control over what an agent can access. - **Strong fit for regulated industries** like banking and healthcare. **Collibra cons:** - **Heavyweight and enterprise-priced.** - **Overkill for fast technical discovery** if formal governance is not your driver. - **Closed source.** ### Secoda {#secoda} **Best for:** teams that want an AI-first hosted catalog without running infrastructure. Secoda is a hosted catalog built around AI from the start, with MCP support and a polished assistant. **Secoda pros:** - **Native, hosted MCP.** Tools like Claude and Cursor connect to trusted metadata including lineage, glossary terms, documentation and SQL context. - **Governed access.** Connections authenticate against workspace permissions. - **Easy to connect** from Cursor, Claude Desktop, VS Code or JetBrains. **Secoda cons:** - **Closed source**, no self-hosting. - **Commercial SaaS** with the usual per-seat considerations. ### Amundsen {#amundsen} **Best for:** existing users running it for search-led discovery. Amundsen, originally from Lyft, helped define modern data discovery with strong search over a metadata graph. **Amundsen pros:** - **Strong search-led discovery** over a graph backed by Neo4j or Atlas and Elasticsearch. - **REST API and lineage** rendered in the UI. - **Apache 2.0 licensed.** **Amundsen cons:** - **No first-class MCP.** There is a community MCP server (the unofficial [`amundsen-mcp`](https://github.com/BrianLondon/amundsen-mcp) project), but nothing maintained by the project itself, so exposing Amundsen to agents means relying on a third-party tool or building your own. - **Slower development pace** relative to OpenMetadata and DataHub. - **Multi-service deployment.** ### Apache Atlas {#apache-atlas} **Best for:** Hadoop and Cloudera estates that already depend on it. Apache Atlas is the metadata and governance backbone of the Hadoop world, with deep hooks into Hive, Spark and the rest of that ecosystem. **Apache Atlas pros:** - **Strong lineage and governance** within the Hadoop domain. - **REST API** for programmatic access. - **Apache 2.0 licensed**, and the metadata you likely already have if you run Hadoop. **Apache Atlas cons:** - **No first-class AI or MCP support.** A community MCP server exists (the unofficial [`apache-atlas-mcp`](https://github.com/DanMeon/apache-atlas-mcp) project), but there is nothing official, so agent access means relying on a third-party tool or wrapping the API yourself. - **Heavy, multi-component stack** (JanusGraph, HBase, Solr, Kafka) that can take months to stand up. - **Wrong starting point for greenfield AI context work.** --- ## The verdict: which data catalog is best for AI agents? {#verdict} **For most teams standing up an AI context layer in 2026, Marmot is the strongest open source starting point.** Native MCP is no longer the differentiator. Most catalogs here ship it now, so connectivity is close to solved. What separates them is everything around it: how much of your stack the catalog can see, how much infrastructure you have to run to keep that context flowing, and whether the metadata it serves is governed and current. Marmot is built around those three things. You get native MCP and governed, vendor-neutral context from a single Go binary on Postgres, populated by plugins or by the Terraform and Pulumi you already write, with no Kafka, graph store, search cluster or platform team to keep it alive. It scales to large catalogs on the same Postgres, with optional Elasticsearch for search at scale, so it is a starting point you do not outgrow. If you need the widest connector library out of the box, a fully managed hosted platform or vendor-held compliance certifications such as SOC 2 and HIPAA, the managed and enterprise options covered above are the better fit. For most teams, Marmot gives you a governed, agent-ready context layer with the least to run, and it is the one to try first. --- ## Frequently asked questions {#faq} ### What is an AI context layer? {#faq-what-is} An AI context layer is the governed source of metadata that an AI agent queries at runtime to understand a data landscape. It exposes schemas, ownership, lineage, tags and business glossary terms through an interface an agent can call, usually MCP or a REST API. In 2026 this is the primary job of a data catalog, because the agent, not just the human, is now the consumer. ### Does my data catalog need MCP support? {#faq-need-mcp} If you want AI assistants like Claude, Cursor or ChatGPT to read your catalog directly, MCP is the path of least resistance. It is a standard interface those tools already understand, so you avoid writing and maintaining a custom integration per assistant. A catalog with a clean REST API can still be wrapped in MCP yourself, but native MCP means there is nothing extra to build or run. ### MCP vs API for AI agents: which should a catalog expose? {#faq-mcp-vs-api} Both, and they serve different callers. A REST API is best for deterministic automation and code you control, where you know exactly which endpoint to call. MCP is best for AI assistants, because the tools are described to the model and it chooses which to call from natural language. A good catalog exposes a stable API and an MCP server that wraps it, so the same governed metadata is reachable either way. ### Can I expose a data catalog to Claude or Cursor? {#faq-claude-cursor} Yes. Any catalog with an MCP server can be connected to Claude Desktop, Claude Code, Cursor, Cline and other MCP clients. You point the client at the catalog's MCP endpoint and authenticate with an API key. The assistant then queries assets, ownership and lineage in natural language, scoped to the permissions of the key you gave it. Marmot's [MCP docs](/docs/MCP) walk through the setup per client. ### Open source or commercial catalog for AI context? {#faq-oss-vs-commercial} Open source catalogs like Marmot, OpenMetadata and DataHub let you self-host and avoid per-seat fees, which matters when an agent makes far more queries than a human. Commercial platforms like Atlan, Collibra and Secoda offer hosted MCP, broad managed connectors and enterprise governance out of the box. The split is the usual one: control and cost against managed breadth and support. ### Why does governed context matter more for agents than for humans? {#faq-governed-context} A human reading a catalog applies judgement and notices when something looks stale. An agent takes the metadata at face value and acts on it. If the catalog hands back raw, uncertified or out of date context, the agent produces confident, wrong output. Governed context, with ownership, certification status and access controls enforced at query time, is what keeps an agent from hallucinating on top of bad metadata. ### Which data catalog is best for AI agents in 2026? {#faq-best-catalog} For most teams in 2026, Marmot is the best open source data catalog for AI agents. It gives you native MCP and governed, vendor-neutral context from a single Go binary on Postgres, with no Kafka, graph store or search cluster to run. OpenMetadata and DataHub fit teams that need the widest connector coverage and can run more infrastructure. Atlan, Collibra and Secoda fit enterprises that want a fully managed, hosted context layer. Marmot is the fastest path to an AI context layer you can actually run and keep current. --- The connectivity problem is close to solved. The catalog that wins as your AI context layer is the one that covers your stack, serves governed and current metadata, and does not cost you a platform team to keep running. Pick on those terms. - **Docs:** [marmotdata.io/docs](/docs/introduction) - **GitHub:** [github.com/marmotdata/marmot](https://github.com/marmotdata/marmot) --- ## Data Governance for the AI Era: A 2026 Guide [← All resources](/resources) Data governance is the set of owners, policies and controls that decide who can see and use what data, and on what terms. For years it was treated as paperwork. In 2026 it is the thing that decides whether an AI agent acting on your data is safe or dangerous. This guide explains what data governance is, what it covers, how it differs from data management, why it matters more once agents are involved, and the practices that make it work. ## What is data governance? {#what-is} **Data governance is the practice of managing the ownership, access, quality and policy around your data so it stays trustworthy and is used correctly.** It answers four questions for every asset: who owns it, who can use it, what it means and what rules apply to it. It is not a single product or a one-off project. It is an ongoing discipline, usually applied through a [data catalog](/resources/data-catalog) that already holds the ownership, classification and lineage governance depends on. Done well, governance is mostly invisible: people and agents get the data they are allowed to use, with the context to use it correctly, and nothing else. ## What does data governance cover? {#what-it-covers} Good governance covers a handful of things, and the value comes from doing them together. - **Ownership.** Every asset has a clear owner and contact, so people and agents know who is responsible and where to escalate. - **Access control.** Permissions enforced at query time, so a caller sees only what its credentials allow, never a raw dump of everything. - **Policies.** Rules for classification, retention and handling of sensitive or regulated data. - **Quality and lineage.** Freshness, certification and how data flows, so a consumer can judge what to trust and what a change will break. - **Auditability.** A record of who accessed what, which matters far more once agents act autonomously. ## Why data governance matters in 2026 {#why} **The stakes rose the moment agents started acting on data.** A person reading a catalog applies judgement and notices when something looks stale or out of bounds. An agent does not. It queries, takes the answer at face value and acts, so the controls have to live in the data layer, not in a human's head. That changes governance from documentation into enforcement. If access is not scoped, an agent can read data it should never see. If context is not governed, it can present a deprecated or uncertified table as fact and build a confident, wrong answer on top of it. Governance is what turns a catalog from a liability into a safe source of context for autonomous tools. We cover the agent side of this in [the AI context layer guide](/resources/ai-context-layer). ## Data governance vs data management {#vs-management} **Data management makes data available. Data governance makes it safe to use.** The two are often confused, but they answer different questions. Data management is the operational work of storing, moving and processing data: the pipelines, warehouses, lakes and the systems that keep them running. Data governance is the layer of ownership, policy and control on top, deciding who may use that data, for what, and how it is classified and retained. You need both. Management without governance gives you data nobody trusts or can use safely. Governance without management is policy with nothing to apply it to. In practice the catalog is where the two meet: it sits over your managed data and applies governance at the point of access. ## How data governance works with AI agents {#agents} **The key shift is governed context rather than raw access.** A catalog that hands an agent everything is a liability. One that scopes each query to a role, the way [Marmot scopes MCP queries to the API key behind them](/resources/mcp-for-data), gives the agent trusted context without overreach. In practice this means three things. Access is enforced when the agent queries, not assumed from where the data sits. Every response is scoped to the caller's permissions, so an agent never receives more than it is entitled to. And the context it does get carries ownership, certification and quality signals, so it can tell a trusted asset from a deprecated one. That is the difference between an agent that works from facts and one that hallucinates on top of whatever it could reach. ## Data governance best practices {#best-practices} You do not need a heavyweight programme to govern data well. A few practices carry most of the weight: - **Assign an owner to every important asset.** Governance without accountable owners is just documentation. - **Enforce access at query time.** Scope each request to the caller rather than copying data into walled gardens. - **Classify sensitive data and set retention.** Tag regulated data and attach the rules that apply to it. - **Make governance self-serve.** Surface ownership, definitions, lineage and quality in a catalog so people and agents find them without asking. - **Keep an audit trail.** Record who accessed what, so autonomous activity stays accountable. - **Govern agents like users.** Give each agent scoped credentials and treat its access exactly as you would a person's. ## How to choose a governance-ready catalog {#how-to-choose} Because governance is usually applied through a [data catalog](/resources/data-catalog), the catalog you pick decides how well you can enforce it. Look for: - **Access control enforced at query time**, across both the API and any MCP interface, scoped per caller. - **Ownership, classification and lineage as first-class metadata**, not bolted-on fields. - **Governed context for agents**, so an MCP query returns only what the key allows. - **An audit trail** of access for both people and agents. We compare the main catalogs on how they expose governed context to AI in [Data Catalogs as the AI Context Layer](/resources/data-catalogs-for-ai-agents). ## Frequently asked questions {#faq} ### What is data governance? {#faq-what} Data governance is the set of owners, policies and controls that decide who can see and use what data, and on what terms. It covers ownership, access control, classification, retention and auditability, so data stays trustworthy, compliant and used correctly. In 2026 governance also has to apply to AI agents, which query data and act on it the same way a person would. ### What does data governance cover? {#faq-covers} Good data governance covers ownership, so every asset has a responsible owner; access control, so permissions are enforced when data is queried; policies for classification, retention and sensitive data; data quality and lineage, so people and agents can judge what to trust; and auditability, a record of who accessed what. Together these make data safe to use rather than just documented. ### What is the difference between data governance and data management? {#faq-vs-management} Data management is the practice of storing, moving and processing data: pipelines, warehouses and the systems that run them. Data governance is the layer of ownership, policy and control that decides who may use that data and how. Management is about making data available; governance is about making it safe and trustworthy to use. You need both, and a data catalog is where governance is usually applied. ### Why does data governance matter for AI agents? {#faq-agents} An AI agent queries data and acts on the result without a human checking each step, so it takes whatever it is given at face value. If access is not scoped, the agent can read data it should not. If context is not governed, it can cite a deprecated or uncertified table as fact. Governance enforced at query time is what keeps an agent inside its lane and stops it acting confidently on the wrong data. ### What are data governance best practices? {#faq-best-practices} Start by assigning a clear owner to every important asset. Enforce access control at query time rather than by copying data into walled gardens. Classify sensitive data and attach retention rules. Make ownership, definitions and quality visible in a catalog so people and agents can self-serve. Keep an audit trail of access. And scope every agent and API query to the caller's permissions, so nothing returns a raw dump of the whole estate. ### Do I need a data governance tool? {#faq-tool} Most teams apply governance through a data catalog rather than a standalone tool. The catalog already holds ownership, classification, lineage and quality, and a good one enforces access control at the point a person or agent queries it. If you run AI agents against your data, a catalog that scopes each query to the caller is close to essential, because that is where governance is actually enforced. ## Related {#related} - [What Is a Data Catalog?](/resources/data-catalog) - [What Is an AI Context Layer?](/resources/ai-context-layer) - [MCP for Data: Connecting AI Agents to Your Catalog](/resources/mcp-for-data) - [Data Catalogs as the AI Context Layer: A 2026 Comparison](/resources/data-catalogs-for-ai-agents) --- ## Data Quality for AI Agents: A 2026 Guide [← All resources](/resources) Data quality is the degree to which your data is accurate, fresh, complete and trustworthy enough to act on. It has always mattered. What changed in 2026 is who consumes it: an AI agent takes your data at face value, so poor quality no longer just misleads a person, it drives an autonomous action. This guide explains what data quality is, the dimensions it breaks into, the tools that measure it, how it differs from data observability, and why it is the foundation an AI context layer is built on. ## What is data quality? {#what-is} **Data quality is how fit your data is for the purpose it will be used for.** It is not an abstract score. It is a practical judgement: can someone, or something, rely on this dataset to make the decision in front of them. Quality is tracked across several dimensions and surfaced through signals: test results, freshness timestamps, and certification that says an asset is trusted and maintained. Those signals are themselves metadata, which means a [data catalog](/resources/data-catalog) can carry them alongside the schema and ownership, and expose them to whoever, or whatever, is about to use the data. ## Data quality dimensions {#dimensions} Quality breaks into a handful of dimensions. A dataset can be strong on some and weak on others, which is why it is worth tracking each one rather than collapsing it to a single number. - **Accuracy.** The data reflects reality. - **Completeness.** No missing rows, columns or relationships that matter. - **Consistency.** It agrees with itself and with other systems. - **Timeliness.** It is fresh enough for the decision at hand. - **Validity.** It conforms to the expected formats, types and rules. - **Uniqueness.** Records are not duplicated. ## Why data quality matters more for AI agents in 2026 {#why} **Agents raise the bar because they remove the human safety net.** A person reading a dashboard applies judgement and notices when a number looks off. An agent does not. It reads the metadata and the data, takes both at face value, and acts, so stale or uncertified context turns straight into confident, wrong output. That makes quality signals part of the context an agent needs, not a back-office concern. An agent that can see "this table is certified and updated hourly" will prefer it over a forgotten copy. An agent that cannot tell the difference will happily build an answer on the wrong one. Quality is the foundation the rest of the [AI context layer](/resources/ai-context-layer) sits on. ## Data quality tools {#tools} **Most teams measure quality with a testing or observability tool, then surface the results in a catalog.** The common options: - **Great Expectations.** An open source framework that validates data against declared "expectations", for example that a column is never null or falls within a range. - **dbt tests.** Built into dbt, these check models inside a project for conditions like uniqueness, not-null and referential integrity as part of the build. - **Soda.** Runs data quality checks written in its own check language, on a schedule or in a pipeline. - **Data observability platforms.** Commercial tools such as Monte Carlo, Bigeye and Anomalo add automated monitoring and anomaly detection across pipelines, catching freshness, volume and schema issues without hand-written tests. These tools produce the signals. A [data catalog](/resources/data-catalog) is where those signals become visible context, so the result of a test or a freshness check is attached to the asset and readable by people and agents alike. ## Data quality vs data observability {#vs-observability} **Data quality is the property you want. Data observability is one way you keep watch on it.** Quality asks "is this data fit to use". Observability is the practice of automatically monitoring pipelines and tables for freshness, volume, schema and anomaly problems, so issues surface before they reach a consumer. The two are complementary. Tests assert the conditions you already know to check. Observability catches the problems you did not predict. Certification records a human judgement that an asset is trusted. All three feed the same picture: how much can this data be relied on, expressed as signals a catalog can carry. ## How to improve data quality {#best-practices} You do not fix quality by buying a tool. You fix it with a few habits the tool supports: - **Define what good looks like** for your most important datasets, then test for it. - **Test where it counts** with Great Expectations, dbt tests or Soda, rather than trying to cover everything at once. - **Monitor freshness and anomalies** with observability where the cost of a silent failure is high. - **Assign owners**, so a failing check has someone accountable for it. - **Certify trusted datasets**, and let that certification be visible. - **Surface every signal in a catalog**, so people and agents can see at a glance what to rely on and what to avoid. ## Quality as context for agents {#agents} **Certification and freshness are themselves metadata an agent should be able to read.** A catalog that surfaces "this table is certified and updated hourly" lets an agent prefer trusted sources and steer around deprecated ones. Without it, the agent cannot tell a golden table from a forgotten one, and every quality problem in your stack becomes a potential agent mistake. This is why quality belongs in the same governed context an agent queries for schema and ownership. When freshness, test results and certification travel with the asset, an agent has what it needs to choose the right data, not just any data that matches the question. ## Frequently asked questions {#faq} ### What is data quality? {#faq-what} Data quality is the degree to which your data is accurate, complete, consistent, fresh and trustworthy enough to act on. It is measured across dimensions like accuracy, completeness, timeliness and validity, and surfaced through signals such as test results, freshness and certification. In 2026 it matters more than ever, because AI agents act on data at face value rather than applying human judgement. ### What are the dimensions of data quality? {#faq-dimensions} The common dimensions are accuracy, whether the data reflects reality; completeness, whether anything is missing; consistency, whether it agrees across systems; timeliness or freshness, whether it is current enough; validity, whether it conforms to expected formats and rules; and uniqueness, whether records are duplicated. A dataset can be strong on some and weak on others, which is why quality is tracked per dimension rather than as a single score. ### What is the difference between data quality and data observability? {#faq-vs-observability} Data quality is the property you care about: is this data fit to use. Data observability is how you monitor for it, automatically watching pipelines and tables for freshness, volume, schema and anomaly issues so problems surface before they reach a consumer. Quality is the goal, observability is one way to keep an eye on it. Tests, observability and certification all feed the same picture of how trustworthy an asset is. ### What are the best data quality tools? {#faq-tools} Common open source options include Great Expectations, which validates data against declared expectations, dbt tests, which check models inside a dbt project, and Soda, which runs checks written in its own check language. Commercial data observability platforms such as Monte Carlo, Bigeye and Anomalo add automated anomaly detection across pipelines. These tools produce the signals; a data catalog is where those results become visible context for people and agents. ### Why does data quality matter for AI agents? {#faq-agents} A human reading a dashboard applies judgement and notices when a number looks wrong. An AI agent takes the metadata and data at face value and acts on it, so stale or uncertified context turns straight into confident, wrong output. Quality signals like freshness and certification let an agent prefer trusted sources and avoid deprecated ones, which is the difference between an agent that works from reliable data and one that does not. ### How do you improve data quality? {#faq-improve} Start by defining what good looks like for your most important datasets, then test for it with a tool like Great Expectations, dbt tests or Soda. Monitor freshness and anomalies with observability where it pays off. Assign owners so issues have someone accountable, certify the datasets that are trusted, and surface all of those signals in a catalog so people and agents can see at a glance what to rely on. ## Related {#related} - [What Is a Data Catalog?](/resources/data-catalog) - [Data Governance](/resources/data-governance) - [What Is an AI Context Layer?](/resources/ai-context-layer) - [Data Catalogs as the AI Context Layer: A 2026 Comparison](/resources/data-catalogs-for-ai-agents) --- ## Marmot Resources: Data Catalogs, Governance and AI Context ## Browse by topic {#topics} ## Latest {#latest} --- ## Marmot vs Atlan: AI Context Layer Comparison (2026) [← All resources](/resources) How Marmot and Atlan compare as AI context layers: an open source catalog you self-host as a single Go binary versus Atlan's fully managed, enterprise SaaS context layer. Both expose governed metadata to AI agents over a native MCP server, so on the agent-facing capability they are closer than they look. The real decision is a different one: whether you want to own and run the catalog yourself, or hand hosting, scale and compliance to a vendor. This page compares them on exactly that. For the full field, see [the data catalog AI context layer comparison](/resources/data-catalogs-for-ai-agents). --- ## At a glance {#at-a-glance} --- ## Hosting and footprint {#hosting} **This is the clearest difference between the two.** Marmot is open source and self-hosted: a single Go binary that needs nothing but Postgres. You run it in your own infrastructure, your metadata never leaves it, and there is no vendor in the data path. You can run it on a small VM or scale it to zero on serverless. Atlan is a fully managed SaaS platform. There is nothing for you to deploy, patch or scale, because the vendor handles hosting, upgrades and uptime. That is a genuine advantage if you would rather not run a catalog at all. The trade is the usual one for managed software: less control over where the data sits, and a commercial relationship instead of a binary you own. If self-hosting and data residency matter to you, Marmot fits; if you want the catalog handed to you as a service, Atlan does. --- ## MCP and AI context {#mcp} **Both serve context to agents over a native MCP server, so neither makes you bolt on a third-party package.** What differs is where the server runs. Marmot's MCP server is part of the binary, so the moment Marmot is running it is already an AI context layer. It exposes three focused tools: `discover_data` for natural language and qualified-identifier lookups with lineage traversal, `find_ownership` for "who owns this", and `lookup_term` for glossary definitions. Every query runs with the permissions of the API key behind it, in infrastructure you control. Atlan hosts its MCP server as part of the managed platform. It exposes search, lineage traversal and metadata operations to tools like Claude, Cursor, ChatGPT and Gemini, with access governed by Atlan's roles and policies. It is well-integrated and there is nothing for you to run, with the trade-off that the context flows through the vendor's hosted service rather than your own. --- ## CLI and tooling {#cli} **Marmot's command line and developer tooling are broader and openly available.** The `marmot` CLI covers search, lineage, glossary and ownership from the terminal, with OAuth or API-key authentication. On top of that Marmot ships a packaged agent Skill, a ready-made instruction set that teaches an assistant how to drive the catalog over the CLI, REST API or MCP without bespoke wiring. Together with native MCP, an agent can work a Marmot catalog the moment it is installed. Atlan's CLI is in closed preview and focused on data contracts and limited metadata sync, available through your account team rather than as a public download. Both offer SDKs, and Marmot's coverage is wider in the open: fully featured Go, TypeScript and Python SDKs, against Python and Java for Atlan, with an experimental Go SDK. It is part of a broader pattern. Between plugins with YAML ingestion through the CLI, a Kubernetes-native operator, Terraform and Pulumi providers, three SDKs, a REST API and MCP, Marmot gives you an unusually large set of first-party integration paths, all open source, which is how a smaller plugin library still reaches most of a stack. --- ## Governed context and compliance {#governed-context} **For agents, governance is not a nice-to-have.** An agent takes whatever metadata it retrieves at face value and acts on it, so the context has to be scoped and trustworthy or the agent confidently acts on the wrong thing. Marmot runs every MCP and API query with the permissions of the API key behind it, so an agent sees only what that key is allowed to see, never a raw dump of the whole catalog. Because you self-host, your governance and compliance posture is yours to define inside your own infrastructure. Atlan enforces access through enterprise roles and policies, and its MCP server respects them. Where Atlan has a clear edge is vendor-held compliance: it carries certifications such as SOC 2 Type II, ISO 27001, HIPAA and GDPR, audited and maintained by the vendor. If you need a provider to hold those certifications for you rather than managing the controls yourself, that is a real reason to choose Atlan. --- ## Connectors and coverage {#connectors} **Atlan ships a large managed connector library with nothing for you to deploy.** Its hosted connectors span warehouses, databases, dashboards and pipelines, and because the platform is managed, keeping them running is the vendor's job rather than yours. Marmot ships around 28 plugins in a fast-growing ecosystem. For anything without a plugin yet, Marmot's official Terraform and Pulumi providers (`marmot_asset`, `marmot_lineage`) populate assets and lineage straight from the infrastructure you already define, so a source still lands in the catalog from code you are writing anyway. The trade is real: if you want the most of your stack catalogued by a managed service with no work, Atlan leads on out-of-the-box breadth. If you provision with Terraform or Pulumi, the gap closes quickly, and you keep the whole thing in your own infrastructure. --- ## Lineage {#lineage} Both expose lineage to agents rather than just rendering it in a UI, and both hold large lineage graphs without trouble. Marmot serves lineage through MCP (`discover_data`), the `marmot` CLI, its Go, TypeScript and Python SDKs and a REST API, answers "what feeds this, and what breaks if I change it" for agents and humans, and stores the graph in Postgres alongside the rest of the catalog. Atlan offers column-level lineage and active metadata as part of its managed platform, which is the thing to reach for if you need field-level impact analysis maintained for you. Both store lineage at scale; the difference is the shape of the query surface, not how much either can hold. --- ## Cost and query economics {#cost} **This is where the open source model tells, especially for agents.** Marmot is MIT licensed and self-hosted, so there are no per-seat or per-query fees. You pay for the infrastructure it runs on and nothing else. That matters because an agent issues far more queries than a person, and usage-based pricing is hard to predict once agents are doing real work against the catalog. Atlan is priced as an enterprise SaaS agreement, typically per seat or per usage. For that you get a managed service, support and the compliance posture above, which is a fair trade if you would rather buy the capability than run it. The point worth scrutinising is how the pricing behaves under agent workloads, where query volume is high and growing. --- ## Which should you choose? {#which-to-choose} **Choose Marmot if:** - You want an open source catalog you self-host and fully control. - You want native MCP and governed context from a single binary, with no per-seat fees as agents scale up their queries. - You would rather keep metadata in your own infrastructure than route it through a vendor. - You provision with Terraform or Pulumi and want catalog-as-code. - You want a catalog an agent can use out of the box, through native MCP, a full CLI and a packaged Skill. **Choose Atlan if:** - You want a fully managed platform with nothing to run or maintain. - You need vendor-held compliance certifications such as SOC 2 Type II, ISO 27001 and HIPAA. - You have the budget for an enterprise SaaS agreement and want a large managed connector library with vendor support. For most teams standing up an AI context layer in 2026, Marmot is the faster path to a governed, agent-ready catalog you own and run yourself, with no per-seat fees as agents scale up their queries. Atlan is the stronger choice when you want a fully managed platform and vendor-held compliance certifications, and are happy to buy that as a service. --- ## Frequently asked questions {#faq} ### Is Marmot an open source Atlan alternative? {#faq-alternative} Yes. Both expose governed metadata to AI agents over a native MCP server, so both work as an AI context layer. The difference is the model. Marmot is open source and MIT licensed, runs as a single Go binary on Postgres and is self-hosted, so you own the stack and pay no per-seat fees. Atlan is a proprietary, fully managed SaaS platform. Marmot's edge is control, footprint and cost. Atlan's is a managed service with vendor-held compliance certifications. ### What is the difference between Marmot and Atlan? {#faq-difference} Marmot is an open source catalog you run yourself: one Go binary on Postgres with native MCP, a full CLI, three SDKs and catalog-as-code. Atlan is a fully managed enterprise SaaS platform with a hosted MCP server, a large managed connector library and vendor-held certifications such as SOC 2 Type II, ISO 27001 and HIPAA. The decision is whether you want to own and run the catalog, or buy it as a managed service. ### Is Atlan open source? {#faq-open-source} No. Atlan is a proprietary, commercial SaaS platform, priced per enterprise agreement. Marmot is open source under the MIT licence, so you can self-host it for free, inspect the code and avoid per-seat or per-usage fees. If you need an open source catalog you can run in your own infrastructure, Marmot is the closer fit. If you want a managed platform and are comfortable with commercial licensing, Atlan is built for that. ### Does Atlan support MCP, and how does it compare to Marmot? {#faq-mcp} Both have a native MCP server. Atlan hosts its MCP server as part of the managed platform, exposing search, lineage and metadata operations to tools like Claude, Cursor, ChatGPT and Gemini, governed by its policies. Marmot's MCP server is built into the binary, so the catalog is agent-ready the moment it starts and every query is scoped to the API key behind it. The capability is similar; the difference is self-hosted in your own infrastructure versus hosted by the vendor. ### Is Marmot cheaper than Atlan for AI agents? {#faq-cost} For most teams, yes. Marmot is MIT licensed and self-hosted, so there are no per-seat or per-query fees. That matters because an agent issues far more queries than a person, and per-usage pricing is hard to predict under agent workloads. Atlan is enterprise-priced as a managed service, so you trade that cost for hosting, scale and support you do not have to run yourself. ### Which is better for AI agents in 2026? {#faq-which-better} For most teams standing up an AI context layer, Marmot is the faster path: native MCP, governed context and a full CLI from one binary you own, with no per-seat fees as agents scale up their queries. Atlan is the better fit when you want a fully managed platform with nothing to run and need vendor-held compliance certifications such as SOC 2 Type II, ISO 27001 and HIPAA. --- - **Docs:** [marmotdata.io/docs](/docs/introduction) - **GitHub:** [github.com/marmotdata/marmot](https://github.com/marmotdata/marmot) --- ## Marmot vs DataHub: AI Context Layer Comparison (2026) [← All resources](/resources) How Marmot and DataHub compare as AI context layers: native MCP on a single binary versus an official MCP server on a Kafka, graph and Elasticsearch stack. Both are open source data catalogs, both expose metadata to AI agents over the Model Context Protocol, and both serve lineage to tools like Claude and Cursor. The real difference is what you have to run to get there, and how much of your stack each one can see. This page compares them on exactly that. For the full field, see [the data catalog AI context layer comparison](/resources/data-catalogs-for-ai-agents). --- ## At a glance {#at-a-glance} --- ## Deployment and footprint {#deployment} **This is the clearest difference between the two.** Marmot is a single Go binary that needs nothing but Postgres. There is no message bus, no graph database and no required search cluster, with Elasticsearch available only if you want it. You can run it on a small VM or scale it to zero on serverless. DataHub's full deployment leans on Kafka for its metadata change log, a graph store for relationships and Elasticsearch for search. That architecture suits streaming metadata, but it is several stateful services to provision, secure and keep healthy. Marmot reaches large catalogs on Postgres without any of it, so this is a difference in operational complexity, not in how much either tool can hold. For a small team it is the difference between running one process and running a platform. --- ## MCP and AI context {#mcp} **Both serve context to agents over MCP, but they package it differently.** Marmot's MCP server is part of the binary, so the moment Marmot is running it is already an AI context layer. It exposes three focused tools: `discover_data` for natural language and qualified-identifier lookups with lineage traversal, `find_ownership` for "who owns this", and `lookup_term` for glossary definitions. Every query is scoped to the permissions of the API key behind it. DataHub ships an official MCP server, published as a separate package by Acryl. It lets agents search assets, traverse lineage, inspect schemas and generate SQL through Cursor, Claude Desktop, Windsurf and others, and it has real production use behind it, including Block's Goose agent. The trade-off is operational: it is one more component to deploy and version against the platform, rather than something built into the core. --- ## CLI and tooling {#cli} **Both ship a full command line interface, which puts them ahead of catalogs that offer only an ingestion or admin utility.** DataHub's `datahub` CLI ingests metadata from YAML recipes and lets you get, update and explore entities from the terminal. Marmot's `marmot` CLI covers search, lineage, glossary and ownership, with OAuth or API-key authentication. Both also ship SDKs, and this is one of Marmot's quieter strengths. Marmot has fully featured Go, TypeScript and Python SDKs, where DataHub offers Python and Java. That breadth is part of a wider point: between plugins with YAML ingestion through the CLI, a Kubernetes-native operator, Terraform and Pulumi providers, three SDKs, a REST API and MCP, Marmot gives you an unusually large set of first-party integration paths to get data in and out. Marmot adds one more thing DataHub does not: a packaged agent Skill. It is a ready-made instruction set that teaches an assistant how to drive Marmot over the CLI, REST API or MCP, so an agent can work the catalog without bespoke wiring. Together with native MCP, it means Marmot is usable by an agent the moment it is installed, not after you stand up and connect a separate server. --- ## Governed context {#governed-context} **For agents, governance is not a nice-to-have.** An agent takes whatever metadata it retrieves at face value and acts on it, so the context has to be scoped and trustworthy or the agent confidently acts on the wrong thing. Marmot runs every MCP and API query with the permissions of the API key behind it, so an agent sees only what that key is allowed to see, never a raw dump of the whole catalog. DataHub enforces access through policies and access controls across the platform. Both can keep an agent inside its lane. The difference is shape: Marmot's governance is one access layer in one process, where DataHub's spans the wider platform and its services. --- ## Connectors and coverage {#connectors} **DataHub has the broader pre-built connector ecosystem today; Marmot covers the rest with catalog-as-code.** DataHub's integration library is extensive, covering a wide range of warehouses, orchestrators and BI tools. Marmot ships around 28 plugins in a fast-growing ecosystem. For anything without a plugin yet, Marmot's official Terraform and Pulumi providers (`marmot_asset`, `marmot_lineage`) populate assets and lineage straight from the infrastructure you already define, so a source still lands in the catalog from code you are writing anyway. DataHub leans on YAML ingestion recipes and its SDK instead. If raw pre-built connector count is your deciding factor, DataHub leads. If you provision with Terraform or Pulumi, the gap closes quickly. --- ## Lineage {#lineage} Both expose lineage to agents rather than just rendering it in a UI, and both hold large lineage graphs without trouble. Marmot serves lineage through MCP (`discover_data`), the `marmot` CLI, its Go, TypeScript and Python SDKs and a REST API, answers "what feeds this, and what breaks if I change it" for agents and humans, and stores the graph in Postgres alongside the rest of the catalog. DataHub adds column-level lineage and a GraphQL surface for traversing it, which is the thing to reach for if you need field-level impact analysis across many sources. Both store lineage at scale; the difference is the shape of the query surface, not how much either can hold. --- ## Which should you choose? {#which-to-choose} **Choose Marmot if:** - You want native MCP and governed context with the smallest possible footprint. - You would rather run one binary on Postgres than a Kafka and search stack. - You provision with Terraform or Pulumi and want catalog-as-code. - You want a catalog an agent can use out of the box, through native MCP, a full CLI and a packaged Skill. - You value simplicity and fast setup over breadth. **Choose DataHub if:** - You need a specific source DataHub already ships a connector for and would rather not provision it as code. - You want a GraphQL metadata API or event and streaming-based ingestion as first-class building blocks. - You have the resource to manage Kafka, a graph store, Elasticsearch and the rest of the stack it expects. For most teams standing up an AI context layer in 2026, Marmot is the faster path to a governed, agent-ready catalog, and it scales to large catalogs without the extra infrastructure. DataHub is the stronger choice when you already run its stack or want its broad integration ecosystem. --- ## Frequently asked questions {#faq} ### Is Marmot a DataHub alternative? {#faq-alternative} Yes, for teams that want an open source data catalog and AI context layer without DataHub's infrastructure. Marmot runs as a single Go binary on Postgres with a built-in MCP server, where DataHub expects Kafka, a graph store and Elasticsearch. Marmot scales to large catalogs on Postgres and is far lighter to run. DataHub's edge is a broader pre-built integration ecosystem and column-level lineage. ### Does DataHub need Kafka? {#faq-kafka} A full DataHub deployment uses Kafka for its metadata change log, plus a graph store and Elasticsearch for search. That architecture suits streaming metadata but is heavier to run and maintain than a single-process catalog. Marmot avoids it entirely, scaling to large catalogs on Postgres alone, so the difference is operational complexity rather than capability. ### Which has better MCP support, Marmot or DataHub? {#faq-mcp} Both expose metadata to AI agents over MCP. Marmot's MCP server is built into the binary, so there is nothing extra to deploy. DataHub's MCP server is an official but separate package you run alongside the platform. Marmot wins on simplicity. DataHub's server exposes a broader surface, including SQL generation across a larger ecosystem. ### Do Marmot and DataHub both have a CLI? {#faq-cli} Yes. Both ship a full command line interface, which sets them apart from catalogs that offer only an ingestion or admin utility. DataHub's `datahub` CLI ingests from YAML recipes and lets you get, update and explore entities. Marmot's `marmot` CLI covers search, lineage, glossary and ownership. Marmot also ships a packaged agent Skill, a ready-made instruction set that lets an assistant drive the catalog over the CLI, REST API or MCP without custom wiring. ### Can I connect Marmot or DataHub to Claude and Cursor? {#faq-claude-cursor} Yes, both expose an MCP server that MCP clients like Claude Desktop, Claude Code and Cursor can connect to. With Marmot the server is built into the binary, so you point the client at the catalog and authenticate with an API key. With DataHub you deploy and connect its separate MCP server package alongside the platform. Either way the assistant can then query assets, ownership and lineage in natural language. ### Which is better for AI agents in 2026? {#faq-which-better} For most teams standing up an AI context layer, Marmot is the faster path: native MCP, governed context and vendor-neutral coverage from one binary on Postgres, and it scales to large catalogs without a Kafka and search stack. DataHub is the better fit when you already run that stack, prefer GraphQL or streaming-based ingestion, or want its broader pre-built integration ecosystem. --- - **Docs:** [marmotdata.io/docs](/docs/introduction) - **GitHub:** [github.com/marmotdata/marmot](https://github.com/marmotdata/marmot) --- ## Marmot vs OpenMetadata: AI Context Layer Comparison (2026) [← All resources](/resources) How Marmot and OpenMetadata compare as AI context layers: native MCP on a single Go binary and Postgres versus the broadest open source connector platform, running on a database, a search cluster and an ingestion framework. Both are open source data catalogs, both ship a native MCP server, and both have positioned themselves as a context layer for AI agents. They overlap more than most pairs in this space, so the real question is footprint against breadth: how much you have to run, against how much the catalog can see out of the box. For the full field, see [the data catalog AI context layer comparison](/resources/data-catalogs-for-ai-agents). --- ## At a glance {#at-a-glance} --- ## Deployment and footprint {#deployment} **This is the clearest difference between the two.** Marmot is a single Go binary that needs nothing but Postgres. There is no separate search cluster and no ingestion service to run, with Elasticsearch available only if you want it. You can run it on a small VM or scale it to zero on serverless. A standard OpenMetadata deployment is several services: a relational database for metadata, Elasticsearch or OpenSearch for search, and an Airflow-based ingestion framework to drive its connectors. It is a capable platform, but it is more to provision, secure and keep healthy. Both hold large catalogs, so this is a difference in operational complexity rather than in how much either tool can store. For a small team it is the difference between running one process and running a platform. --- ## MCP and AI context {#mcp} **Both serve context to agents over a native MCP server, so neither makes you bolt on a separate package.** This is the rare pair where MCP itself is not the differentiator. What differs is what sits behind it. Marmot's MCP server is part of the binary, so the moment Marmot is running it is already an AI context layer. It exposes three focused tools: `discover_data` for natural language and qualified-identifier lookups with lineage traversal, `find_ownership` for "who owns this", and `lookup_term` for glossary definitions. Every query runs with the permissions of the API key behind it. OpenMetadata's MCP server is a first-class part of the platform, with OAuth 2.0 authentication and access governed by its roles and policies. It is well-integrated, but it is one capability of a larger multi-service system you stand up first, rather than something that comes alive with a single process. --- ## CLI and tooling {#cli} **Both offer a command line tool, but they differ in scope.** OpenMetadata's official `metadata` CLI is focused on running ingestion workflows and administration, which fits its ingestion-framework model. Marmot's `marmot` CLI is broader: search, lineage, glossary and ownership from the terminal, with OAuth or API-key authentication. On top of that Marmot ships a packaged agent Skill, a ready-made instruction set that teaches an assistant how to drive the catalog over the CLI, REST API or MCP without bespoke wiring. Together with native MCP, it means an agent can work a Marmot catalog the moment it is installed. Both also offer SDKs, and Marmot's coverage is wider: fully featured Go, TypeScript and Python SDKs, against Python and Java for OpenMetadata. It is part of a broader pattern. Between plugins with YAML ingestion through the CLI, a Kubernetes-native operator, Terraform and Pulumi providers, three SDKs, a REST API and MCP, Marmot gives you an unusually large set of first-party integration paths, which is how a smaller plugin library still reaches most of a stack. --- ## Governed context {#governed-context} **For agents, governance is not a nice-to-have.** An agent takes whatever metadata it retrieves at face value and acts on it, so the context has to be scoped and trustworthy or the agent confidently acts on the wrong thing. Marmot runs every MCP and API query with the permissions of the API key behind it, so an agent sees only what that key is allowed to see, never a raw dump of the whole catalog. OpenMetadata enforces access through roles and policies across the platform, and its MCP server respects them. Both keep an agent inside its lane. The difference is shape: Marmot's governance is one access layer in one process, where OpenMetadata's is part of the wider platform. --- ## Connectors and coverage {#connectors} **This is OpenMetadata's strongest column.** Its connector library is the widest in open source, well past 120 sources spanning warehouses, databases, dashboards, pipelines and messaging, all driven through its ingestion framework. If you want the most of your stack catalogued out of the box with no extra work, OpenMetadata leads here and it is not close. Marmot ships around 28 plugins in a fast-growing ecosystem. For anything without a plugin yet, Marmot's official Terraform and Pulumi providers (`marmot_asset`, `marmot_lineage`) populate assets and lineage straight from the infrastructure you already define, so a source still lands in the catalog from code you are writing anyway. The trade is real: if raw pre-built connector count is your deciding factor, OpenMetadata wins. If you provision with Terraform or Pulumi, the gap closes quickly, and you avoid running an ingestion framework to do it. --- ## Lineage {#lineage} Both expose lineage to agents rather than just rendering it in a UI, and both hold large lineage graphs without trouble. Marmot serves lineage through MCP (`discover_data`), the `marmot` CLI, its Go, TypeScript and Python SDKs and a REST API, answers "what feeds this, and what breaks if I change it" for agents and humans, and stores the graph in Postgres alongside the rest of the catalog. OpenMetadata offers column-level lineage with field-level detail, which is the thing to reach for if you need impact analysis across many sources. Both store lineage at scale; the difference is the shape of the query surface, not how much either can hold. --- ## Which should you choose? {#which-to-choose} **Choose Marmot if:** - You want native MCP and governed context with the smallest possible footprint. - You would rather run one binary on Postgres than a database, a search cluster and an ingestion framework. - You provision with Terraform or Pulumi and want catalog-as-code. - You want a catalog an agent can use out of the box, through native MCP, a full CLI and a packaged Skill. - You value simplicity and fast setup over breadth. **Choose OpenMetadata if:** - You need the widest pre-built connector coverage in open source. - You have the resource to manage a search cluster, an ingestion framework and the rest of the stack it expects. For most teams standing up an AI context layer in 2026, Marmot is the faster path to a governed, agent-ready catalog, with the least to run. OpenMetadata is the stronger choice when the widest out-of-the-box connector coverage is the priority and you can support the services behind it. --- ## Frequently asked questions {#faq} ### Is Marmot an OpenMetadata alternative? {#faq-alternative} Yes. Both are open source data catalogs with a native, built-in MCP server, so both work as an AI context layer out of the box. The difference is footprint. Marmot runs as a single Go binary on Postgres, where OpenMetadata expects a database, a separate Elasticsearch or OpenSearch cluster and an Airflow-based ingestion framework. OpenMetadata's edge is the widest open source connector library. Marmot's is the smallest operational footprint. ### What is the difference between Marmot and OpenMetadata? {#faq-difference} Both ship native MCP and expose governed context to agents. OpenMetadata is the broader platform, with 120+ pre-built connectors and column-level lineage, but it runs as several services. Marmot is lighter: one binary on Postgres, a full CLI and a packaged agent Skill, and catalog-as-code through official Terraform and Pulumi providers for sources without a plugin. ### Does OpenMetadata need Elasticsearch and Airflow? {#faq-dependencies} A standard OpenMetadata deployment uses a relational database for metadata, Elasticsearch or OpenSearch for search, and an Airflow-based ingestion framework to run its connectors. That is several services to provision and maintain. Marmot needs only Postgres, with Elasticsearch optional, so it is materially lighter to run for the same job. ### Which has better MCP support, Marmot or OpenMetadata? {#faq-mcp} Both have a native MCP server built into the product, so neither makes you deploy a separate package. The practical difference is what sits behind it: with Marmot the MCP server is part of one binary, so the catalog is agent-ready the moment it starts. With OpenMetadata the MCP server is one capability of a larger multi-service platform you stand up first. ### Do Marmot and OpenMetadata have a CLI? {#faq-cli} Both offer a command line tool, but they differ in scope. OpenMetadata's official `metadata` CLI is focused on running ingestion workflows and administration. Marmot ships a full `marmot` CLI for search, lineage, glossary and ownership, plus a packaged agent Skill that lets an assistant drive the catalog over the CLI, REST API or MCP without custom wiring. ### Which is better for AI agents in 2026? {#faq-which-better} For most teams standing up an AI context layer, Marmot is the faster path: native MCP, governed context and a full CLI from one binary on Postgres, with the least to run. OpenMetadata is the better fit when you need the widest pre-built connector coverage out of the box and are happy to run and maintain the supporting search and ingestion services. --- - **Docs:** [marmotdata.io/docs](/docs/introduction) - **GitHub:** [github.com/marmotdata/marmot](https://github.com/marmotdata/marmot) --- ## MCP for Data: Connecting AI Agents to Your Catalog [← All resources](/resources) The Model Context Protocol (MCP) is an open standard for connecting AI assistants to external systems, including data catalogs, through a common interface. For data teams it is the cleanest way to let tools like Claude and Cursor read governed context about your stack. This guide explains what MCP is, how it works with a data catalog, how it compares to a plain API, and how to connect an assistant to your catalog. ## What is MCP (Model Context Protocol)? {#what-is} **MCP is an open standard that lets AI assistants discover and call tools exposed by an external system, over one common interface.** Before MCP, wiring an assistant to your metadata meant a bespoke integration per tool. MCP standardises it: the system exposes tools the model can discover and call, so Claude, Cursor, ChatGPT and others reach the same context without custom glue for each. For a [data catalog](/resources/data-catalog), this is what turns it into a live context source for agents rather than a UI people log into. The catalog publishes its capabilities as MCP tools, and any MCP-aware assistant can use them. ## How MCP works with a data catalog {#how-it-works} **A catalog exposes its context as MCP tools, and the assistant calls them on demand.** In practice there are three moving parts: - The catalog runs an **MCP server** that describes its tools, for example search assets, find ownership and traverse lineage. - An **MCP client** inside the assistant connects and authenticates, usually with an API key. - The model **chooses which tools to call** from a natural language request, and the catalog returns context scoped to the key's permissions. So a question like "who owns the orders table and what feeds it" becomes a couple of tool calls against the catalog, answered from live metadata rather than the model's guesses. ## What an MCP server for a catalog exposes {#tools} The exact tools vary by product, but a catalog's MCP server typically offers: - **Search and discovery**, to find assets by name, type or domain. - **Ownership lookup**, to answer "who is responsible for this". - **Lineage traversal**, to follow what feeds an asset and what depends on it. - **Glossary lookup**, to resolve business terms to their agreed meaning. Each call returns context scoped to the caller, so the assistant works with governed data, not a raw export. ## MCP vs API {#mcp-vs-api} **Both matter, and they serve different callers.** A REST API is best for deterministic automation you control, where your code knows exactly which endpoint to hit. MCP is best for AI assistants, because the tools are described to the model and it decides what to call from a natural language request. A good catalog offers both, with the MCP server wrapping the same API and the same governance. That way a pipeline can call the API directly while an assistant reaches the identical context over MCP, and neither sees more than its credentials allow. We go deeper on giving agents context in [AI data engineering](/resources/ai-data-engineering). ## Native MCP vs a separate MCP server {#native-vs-separate} **Some catalogs build MCP into the product, others ship it as a separate package.** With native MCP the server is part of the catalog, so it is available the moment the catalog runs, with nothing extra to deploy or version. A separate MCP server is a distinct component you install and keep in step with the platform. Both speak the same protocol to assistants, so the difference is operational rather than functional: one fewer moving part against an additional service to run. It is one of the axes we compare across tools in [Data Catalogs as the AI Context Layer](/resources/data-catalogs-for-ai-agents). ## How to connect a data catalog to Claude or Cursor {#connect} **If the catalog has an MCP server, connecting an assistant is a short, standard setup.** The shape is the same across clients: - Get an **API key** from the catalog, scoped to what you want the assistant to access. - Point the **MCP client** in Claude Desktop, Claude Code, Cursor or Cline at the catalog's MCP endpoint. - **Authenticate** with the key, and the assistant discovers the catalog's tools. From there the assistant can query assets, ownership and lineage in natural language, scoped to that key. Marmot ships its MCP server in the binary, so there is no separate package to run; the [Marmot MCP docs](https://marmotdata.io/docs/MCP/) walk through the exact steps. ## Frequently asked questions {#faq} ### What is the Model Context Protocol (MCP)? {#faq-what} The Model Context Protocol (MCP) is an open standard for connecting AI assistants to external systems, including data catalogs, through a common interface. The system runs an MCP server that describes the tools it offers, and the assistant's MCP client connects and calls them. It means a tool like Claude or Cursor can reach your data context without a bespoke integration built for each one. ### How does MCP work with a data catalog? {#faq-how} The catalog runs an MCP server that exposes tools such as search assets, find ownership and traverse lineage. An MCP client inside the assistant connects and authenticates, usually with an API key. The model chooses which tools to call from a natural language request, and the catalog returns context scoped to that key's permissions, so the assistant can answer questions about your data without a custom connector. ### What is the difference between MCP and a REST API? {#faq-vs-api} A REST API is best for deterministic automation you control, where your code knows exactly which endpoint to call. MCP is best for AI assistants, because the tools are described to the model and it decides which to call from natural language. They are complementary: a good catalog exposes a stable API and an MCP server that wraps the same governed data, so both callers reach it. ### How do I connect a data catalog to Claude or Cursor? {#faq-connect} If the catalog has an MCP server, you point the MCP client in Claude Desktop, Claude Code, Cursor or Cline at the catalog's MCP endpoint and authenticate with an API key. The assistant then discovers the catalog's tools and can query assets, ownership and lineage in natural language, scoped to the permissions of the key you provided. ### What is the difference between native MCP and a separate MCP server? {#faq-native-vs-separate} Native MCP means the server is built into the catalog, so it is available the moment the catalog runs, with nothing extra to deploy. A separate MCP server is a distinct package you install and version alongside the platform. Both expose the same protocol to assistants; the difference is operational, one fewer moving part with native MCP versus an additional component to run and keep in step. ### Is MCP access to a data catalog secure? {#faq-secure} It can be, when access is scoped at the point of the query. A well-designed catalog authenticates the MCP client with an API key and returns only what that key is permitted to see, never a raw dump of the whole catalog. That means an agent connected over MCP stays inside the same permissions a person with that key would have. ## Related {#related} - [What Is an AI Context Layer?](/resources/ai-context-layer) - [AI Data Engineering: Giving Agents Real Context](/resources/ai-data-engineering) - [Data Governance](/resources/data-governance) - [Data Catalogs as the AI Context Layer: A 2026 Comparison](/resources/data-catalogs-for-ai-agents) --- ## Marmot: Data catalog without the complex infrastructure Data catalogs shouldn't need an entire platform team to run them. Marmot is an open source data catalog that just needs PostgreSQL - no Kafka, no Elasticsearch, no Airflow. A single binary, deployable in minutes, focused on simplicity for everyone. ## Why Marmot? Modern data stacks are fragmented. Assets live across vendors, warehouses, message queues, object storage and APIs. Existing open source catalogs can help - but they come with baggage. Many require external orchestrators, search indexers and message brokers alongside the main application. That means more infrastructure to manage, more things to break and more complexity to debug. - **PostgreSQL only** - no Elasticsearch, no message brokers, no graph databases - **Go-based** - a single binary that runs comfortably on modest infrastructure - **No orchestrator dependency** - ingestion runs directly from the UI or CLI - **Terraform and Pulumi native** - infrastructure-as-code from day one - **Custom query language** - find any asset with precise queries, not just basic filters - **Open source** - MIT licence with enterprise features like SSO included ## Architecture Marmot is built entirely in Go with PostgreSQL being the only external dependency, handling search, job scheduling and metadata storage. It's so lightweight that a full instance runs comfortably on a single $4/month cloud instance. Unlike traditional catalogs that have opinionated ingestion methods, Marmot lets you populate your catalog however you like. The UI supports manual entries and automated discovery via the plugin system. The CLI uses the same plugin system, so you can run ingestion jobs from your Marmot instance or as part of your CI/CD pipelines. Terraform, Pulumi and the REST API are there for infrastructure-as-code workflows and custom integrations. ## Discovery and Lineage Plugins let you get started quickly. PostgreSQL, MySQL, MongoDB, ClickHouse, BigQuery, Kafka, S3, GCS, Azure Blob, Airflow, dbt and more - with the ecosystem growing. Simply fill out the configuration with the required fields and run the job via the UI. You can also run the same plugins via the CLI if you want to keep your ingestion jobs local to your data assets. For everything else, Terraform, Pulumi and the REST API let you document almost anything. Supported plugins such as Airflow and dbt will automatically fill out lineage for assets, allowing you to quickly see the flow of data between your assets. You can also manually link assets in the UI or wire them up with Terraform or Pulumi. You can even capture lineage automatically via OpenLineage. ## Search Marmot has a custom query language that lets you search across everything - assets, glossary terms, data products - all in one place. e.g Find all PostgreSQL tables owned by a specific team: ``` @provider = PostgreSQL AND @type = Table AND @metadata.owner = "Order Management Team" ``` Full-text search, metadata filters, boolean logic, wildcards and range queries. The query language is expressive enough to handle complex searches but simple enough that you'll pick it up in minutes. ## AI Integration Marmot includes a built-in [Model Context Protocol (MCP)](/docs/MCP) server. This lets AI assistants query your catalog using natural language - ask questions like "what tables does the analytics team own?" or "show me the upstream dependencies for user_events" directly in your editor or chat interface. Works with Claude Desktop, Claude Code, Cursor, Cline and other MCP-compatible tools. ## Try It Out Marmot is still early and I'm actively looking for feedback to shape the roadmap. Get involved on GitHub, reach out on Discord or just deploy it and let me know what you think. - **Documentation:** [marmotdata.io/docs](https://marmotdata.io/docs/introduction) - **GitHub:** [github.com/marmotdata/marmot](https://github.com/marmotdata/marmot) --- ## Postgres: One Database to Rule Them All I'm a huge fan of simple software. There's something really satisfying about solving a problem and reducing the number of moving parts. Most of the time, you'll find the tool you already have can do more than you thought. ## The infrastructure tax As a platform engineer, I was genuinely surprised by how much infrastructure existing data catalogs need. Multiple databases, search engines, message queues, workflow orchestration - that's a lot of moving parts before you've even cataloged anything. "But surely big companies need X? They can justify the complexity, right?" Maybe they can soak up the extra cost and complexity, or maybe they're already running this infrastructure for other services, but it doesn't necessarily mean it's required. It's a common trap - reaching for "best practice" tools without questioning if they match your actual scale and requirements. Data catalogs aren't consumer products. They're internal tools used by engineering teams. Even at large organisations, you're looking at hundreds of concurrent users, maybe a couple thousand at most. When the scale ceiling is predictable and modest, why immediately reach for these tools? Whilst building Marmot, I really wanted to see how far I could push Postgres before needing to reach for dedicated search indexers. Turns out, Postgres has a lot more features than most people realise. --- ## What Postgres can do ### Full-text search Postgres has had full-text search since version 8.3 (2008). The core abstraction is `tsvector`, a sorted list of normalised words with positional information. You can weight fields into four priority levels (A highest, D lowest) so for example, matches in a name rank higher than matches in a description: ```sql search_text tsvector GENERATED ALWAYS AS ( setweight(to_tsvector('english', COALESCE(name, '')), 'A') || setweight(to_tsvector('english', COALESCE(mrn, '')), 'A') || setweight(to_tsvector('english', COALESCE(type, '')), 'B') || setweight(to_tsvector('english', array_to_string(providers, ' ')), 'B') || setweight(to_tsvector('english', COALESCE(description, '')), 'C') ) STORED ``` A GIN index on this column lets you query it efficiently. The `websearch_to_tsquery` function handles query parsing with phrases, boolean operators and prefix matching. Ranking uses `ts_rank_cd` which considers density and weight. ### Fuzzy matching Users make typos, also, data assets traditionally don't have very friendly names and will likely include underscores and special characters. The `pg_trgm` extension handles this with trigram similarity. A trigram is three consecutive characters - the PostgreSQL docs note that "a string is considered to have two spaces prefixed and one space suffixed when determining the set of trigrams." So "foobar" becomes `{" f"," fo","foo","oo "," b"," ba","bar","ar "}`. Comparing trigram sets gives you fuzzy matching that handles misspellings and partial matches. ### GIN vs GiST Postgres offers two index types for trigram operations: GiST and GIN. I chose GIN for Marmot after load testing revealed significant performance differences under concurrent load. [The PostgreSQL documentation recommends GIN as the preferred text search index type.](https://www.postgresql.org/docs/current/textsearch-indexes.html) I load tested both approaches under Marmot's read-heavy workload. GIN indexes, which store trigrams in an inverted index structure, showed roughly 3x faster lookups than GiST's balanced tree structure. While GiST indexes are faster for writes and smaller on disk, they were significantly less performant under heavy concurrent read load. For Marmot at least, GIN was the clear winner. The trade-off is slightly slower index updates, which is acceptable since search index updates happen via triggers on entity changes instead of refreshing the index periodically. ### Graph traversal Data lineage is a graph problem.You need to traverse these assets relationships in both directions. Postgres handles this with recursive CTEs: ```sql WITH RECURSIVE upstream AS ( SELECT source_mrn as mrn, -1 as depth FROM lineage_edges WHERE target_mrn = $1 UNION ALL SELECT e.source_mrn, u.depth - 1 FROM lineage_edges e JOIN upstream u ON e.target_mrn = u.mrn WHERE u.depth > -$2 ) CYCLE mrn SET is_cycle USING path SELECT DISTINCT mrn, depth FROM upstream WHERE NOT is_cycle ``` The `RECURSIVE` keyword tells Postgres to iteratively expand the result set - starting with direct dependencies, then repeatedly joining to find dependencies of dependencies. With data pipelines, it can be common for lineage trees to eventually loop around so the `CYCLE` clause detects and marks them automatically. For typical catalog queries (a few dozen assets, 5-10 levels deep), this works well. When rendering massive lineage trees - 250+ assets with depth of 10+ - performance degrades noticeably. The solution for Marmot was to restrict depth in the UI for very large graphs. --- ## How Marmot uses these features ### Search strategy Full-text search and trigram similarity solve different problems. Full-text understands language - stemming, phrases, boolean logic. Trigrams handle typos and partial matches without caring about word boundaries. Marmot uses trigram similarity for its primary search. Why? Data asset names are messy. They're underscore_delimited, dot.separated, or CamelCased. Full-text search handles underscores fine, but struggles with dot.separated names, CamelCase, and abbreviations. Trigrams don't care about delimiters or word boundaries, they just compare character sequences. For structured queries with filters (type, provider, tags), Marmot combines trigram matching with standard SQL predicates. The query planner handles it efficiently in a single round trip. ### Keeping search in sync Marmot maintains a unified search index table that consolidates assets, glossary terms, teams and data products. Each entity type has its own trigger that keeps the index in sync: ```sql CREATE TRIGGER search_index_asset_sync AFTER INSERT OR UPDATE OR DELETE ON assets FOR EACH ROW EXECUTE FUNCTION search_index_asset_trigger(); ``` When an asset changes, the trigger upserts into the search index in the same transaction. The index is always exactly in sync because updates are atomic. [Modern Postgres recommends `GENERATED ALWAYS AS` columns for maintaining tsvector indexes](https://www.postgresql.org/docs/current/textsearch-tables.html#TEXTSEARCH-TABLES-INDEX) - it's simpler and more efficient than triggers. But generated columns only work within a single table; they can't write to separate tables. Since Marmot consolidates multiple entity types into one unified search table, triggers worked well for this use-case. This gives us a denormalised search structure optimized for queries while keeping source tables normalized for writes. For expensive aggregations like facet counts, Marmot maintains a counter cache table that tracks counts by dimension (entity type, asset type, provider, tag). The same triggers that update the search index also update these counters. The downside is that writes become slightly slower since they wait for triggers to complete. For a data catalog with bursty workloads - bulk imports overnight, occasional metadata updates during the day - I found this trade-off is acceptable. --- ## Trade-offs ### What you give up Postgres full-text search isn't Elasticsearch. There are things you give up: - **Scale** - Elasticsearch is built for billions of documents; Postgres full-text search is a bit more modest, however in my testing, it seemed to handle a million assets surprisingly well. - **Simpler field boosting** - four weight levels (A/B/C/D) vs Elasticsearch's numeric boost factors - **No language detection** - you pick your dictionary upfront These are real limitations. Whether they matter depends on your use-case. A data catalog indexes internal metadata - tens of thousands of assets, maybe up to a million for some use-cases, not billions. Users search for table names and column descriptions, not multilingual documents. For this workload, I've found Postgres full-text search to be more than enough. ### Operational simplicity Running Elasticsearch, Neo4j and Kafka means monitoring, tuning, upgrades and capacity planning. That's infrastructure cost and engineering time. Most teams building a data catalog don't have a platform team to run all this. Is Postgres the best tool for search? No, Elasticsearch is better. For graphs? Neo4j wins. But Postgres is good enough for both, and you're already running it. Sometimes the right architecture isn't the one with the best components - it's the one you can actually maintain. --- ## Does it actually work? This all sounds great in theory, but does it actually work beyond 1 user in my local dev environment? ### Test environment I used a dedicated Kubernetes cluster on Hetzner Cloud: - **Cluster**: 7 ARM64 nodes (1 control plane + 6 workers), 4 vCPU and 8GB RAM each - **PostgreSQL**: CloudNativePG with 3 instances (1 primary + 2 read replicas), PgBouncer connection pooling - **Marmot**: 4 replicas spread across workers - **Load generator**: k6 on a dedicated node The database was seeded with 500,000 assets - tables, topics, dashboards, pipelines - with realistic metadata, tags and lineage relationships. k6 simulated 100 concurrent users for 15 minutes, cycling through various patterns including search queries of varying scope and complexity, averaging around 85 requests/second across all endpoints. ### Results ``` ───────────────────────────────────────────────────────────────────── OVERALL HTTP PERFORMANCE ───────────────────────────────────────────────────────────────────── All Requests avg=43.19ms p95=205.32ms p99=481.21ms ───────────────────────────────────────────────────────────────────── ENDPOINT PERFORMANCE ───────────────────────────────────────────────────────────────────── Asset Get avg=7.90ms p95=19.24ms p99=33.66ms Asset Summary avg=5.96ms p95=15.17ms p99=27.36ms Search (Plain) avg=150.99ms p95=452.54ms p99=886.54ms Search (Structured) avg=16.90ms p95=48.96ms p99=101.69ms Search (Empty/Browse) avg=9.50ms p95=20.59ms p99=35.61ms Lineage avg=20.17ms p95=43.69ms p99=61.72ms Metrics Overview avg=11.80ms p95=29.26ms p99=44.65ms Tag Suggestions avg=29.24ms p95=151.49ms p99=305.55ms Field Suggestions avg=14.04ms p95=65.59ms p99=156.24ms ``` Most operations respond in under 20ms. Plain text search is the slowest at ~150ms average - that's the cost of trigram fuzzy matching with 500k assets - but it still feels fast to users. The point isn't that Postgres scales infinitely. I'm sure I could break it with enough load. But for actual data catalog workloads - internal tools used by engineering teams - it handles 100 concurrent users quite easily and I'm sure there's more tuning I could make and more read replicas I could add to keep scaling out! I used CloudNativePG and it made the load test setup really straightforward. These tests ran on modest ARM64 nodes - I'd be interested to see how Marmot performs on managed Postgres services with more resources, if you want to sponsor load testing with some cloud credits, please reach out! --- ## Conclusion Postgres is an awesome tool with a lot of very cool features, [OpenAI recently shared how they run ChatGPT's backend on a single Postgres primary with read replicas](https://openai.com/index/scaling-postgresql/) - serving 800 million users. That said, this approach won't solve all your problems. If you need geo-spatial search, real-time analytics or advanced graph algorithms - specialised tools make sense. The operational complexity pays for itself at that scale. The real bottleneck in data catalogs was never database throughput anyway. It's getting people to actually document their data and keep it up to date! --- ## AI is making assumptions about your data, and getting them wrong If an LLM doesn't have context around your data landscape, it can't help you build new functionality and maintain your existing solutions. It doesn't matter how good the model is. AI assistants are getting really good at writing code. However, a big problem is still understanding your data - what tables exist, who owns them, how they connect, what the business terms actually mean. And right now, most AI tools are completely blind to all of that. The capability to connect AI to external data sources already exists through [MCP (Model Context Protocol)](https://modelcontextprotocol.io). The hard part is having something on the other end that actually covers your entire data landscape. --- ## The problem AI is only as good as what it can access. In most organisations, data knowledge is scattered across Slack threads, stale Confluence pages and READMEs that were accurate months ago. When an AI queries these fragmented sources - or has no source at all - you get confident-sounding wrong answers. And a developer who gets a wrong answer will act on it. Now imagine asking an AI to build a new pipeline - a daily aggregation feeding a customer health dashboard. Without context, it won't know your orders live in Postgres, that there's already a Kafka topic streaming payment events, or that the data-eng team has naming conventions for dbt models. It'll hallucinate table names, guess at schemas and produce something that looks plausible but doesn't fit your stack. You'll spend more time fixing the output than writing it yourself. Give that same AI access to a catalog with real schemas, ownership and lineage - and the output is fundamentally different. --- ## This isn't theoretical OpenAI wrote about [building a bespoke in-house data agent](https://openai.com/index/inside-our-in-house-data-agent) to help their teams explore and reason over their own data platform. They have 600 petabytes of data across 70k datasets - and even they found that simply finding the right table was one of the most time-consuming parts of doing analysis. As they put it: "without context, even strong models can produce wrong results." Their solution was to build multiple layers of context on top of their data - schema metadata, table lineage, human annotations, institutional knowledge - so the agent could actually understand what it was looking at. Even the company building the most capable models in the world found that the model alone wasn't enough. They needed structured, queryable context about their data landscape. That's exactly what a data catalog gives you. --- ## Why vendor neutral matters Real data landscapes don't live in one place. You've got Postgres for application data, Kafka for event streaming, S3 for storage, dbt for transformations, Airflow orchestrating pipelines, BigQuery for analytics and Tableau for dashboards. A source of truth that only covers one vendor's ecosystem is a partial picture. It'll tell you about the BigQuery tables whilst completely missing the Kafka topics that feed them. If your catalog can't see your Postgres tables, your S3 buckets and your Airflow DAGs in the same place, your AI assistant can't either - and you end up relying on a disparate collection of MCP tools that don't mesh well together. --- ## Meeting developers where they work A catalog UI is great for exploring and browsing. But when a developer is authoring a new pipeline or debugging an existing one, they don't want to leave their IDE to look up who owns a table or what feeds a dashboard. AI assistants give them a way to query that same catalog without switching context. It's just another interface to your catalog, one that fits into the workflow developers are already in. That also means the catalog's API matters just as much as its UI - if the metadata behind it is incomplete or stale, those same problems get surfaced right back into the developer's workflow. They end up spending more time correcting the AI's output than they saved by using it in the first place, or worse, they don't notice and ship it. --- ## In the real world You get paged at 2am. Revenue numbers on the executive dashboard look wrong. Without a catalog, you're searching Slack, opening stale Confluence pages that reference deprecated table names and pinging people who are asleep. With a catalog your AI assistant can query, the same incident plays out differently: - **"What tables feed the revenue dashboard?"** - you get the full lineage chain. BigQuery summary table, built by a dbt model pulling from production orders and subscriptions. - **"Who owns the orders pipeline?"** - Order Management team, along with their on-call contact. - **"What does ARR mean here?"** - Annual Recurring Revenue, calculated as the sum of active subscription values normalised to 12 months. --- ## The source of truth is the bottleneck AI capabilities will keep improving and the protocols connecting AI to external data are maturing fast. The "how do I get AI to talk to my data?" problem is effectively solved. The bottleneck is the source of truth. If your AI tools have access to complete, accurate metadata about your data landscape, they become genuinely useful for building and maintaining pipelines. If they don't, you're just adding a layer of confident-sounding guesswork on top of incomplete information. A data catalog that covers your entire stack and exposes it through an API gives your AI tools something real to work with. - **Docs:** [marmotdata.io/docs](/docs/introduction) - **GitHub:** [github.com/marmotdata/marmot](https://github.com/marmotdata/marmot) --- ## Deploy Marmot to Google Cloud Run Marmot ships as a single Go binary and needs nothing but Postgres to run. That suits serverless well, where you pay only while the app is serving traffic. This post walks through deploying it on **Google Cloud Run** with a managed **Cloud SQL for PostgreSQL** database, defined in Terraform. The result scales to zero when idle, no Kubernetes, no sidecars, and no other magic to manage. --- ## What we're building Two managed services, a private network, and the wiring between them: - **Cloud Run** runs the `ghcr.io/marmotdata/marmot` container. It scales to zero, so you only pay when someone is actually using the catalog. - **Cloud SQL for PostgreSQL** stores everything. Marmot needs PostgreSQL 14 or later. It has no public IP and sits on a private network. - **Secret Manager** holds the database password and the encryption key, injected into the container at runtime. Both are generated by Terraform [ephemeral resources](https://developer.hashicorp.com/terraform/language/resources/ephemeral) and written to Secret Manager with [write-only arguments](https://developer.hashicorp.com/terraform/language/resources/ephemeral/write-only), so neither ever lands in Terraform state. Cloud Run and Cloud SQL talk over a fully private path. The database has no public IP at all; Cloud Run joins the same VPC with Direct VPC egress and connects straight to its private IP. --- ## Prerequisites - A Google Cloud project with billing enabled - [Terraform](https://developer.hashicorp.com/terraform/install) 1.11+ (for ephemeral resources and write-only arguments) - [`gcloud`](https://cloud.google.com/sdk/docs/install) authenticated locally: ```bash gcloud auth application-default login ``` --- ## Providers We use the `google` provider for the infrastructure and `random` provider to generate the database password and encryption key, so no secrets are ever written by hand. Set your project here, and the region on the resources below. ```hcl # versions.tf terraform { required_version = ">= 1.11" required_providers { google = { source = "hashicorp/google" version = ">= 6.14" } random = { source = "hashicorp/random" version = ">= 3.7" } } } provider "google" { project = "my-project" } ``` --- ## Enable the APIs ```hcl # apis.tf locals { services = [ "run.googleapis.com", "sqladmin.googleapis.com", "secretmanager.googleapis.com", "compute.googleapis.com", "servicenetworking.googleapis.com", "artifactregistry.googleapis.com", ] } resource "google_project_service" "marmot_apis" { for_each = toset(local.services) service = each.value disable_on_destroy = false } ``` --- ## Private networking The database has no public IP, so it lives on a private VPC. We create the network, a subnet for Cloud Run's Direct VPC egress, and a [private services access](https://cloud.google.com/vpc/docs/private-services-access) range that Cloud SQL's managed service peers into. ```hcl # network.tf resource "google_compute_network" "marmot" { name = "marmot" auto_create_subnetworks = false depends_on = [google_project_service.marmot_apis] } resource "google_compute_subnetwork" "marmot" { name = "marmot" region = "europe-west1" network = google_compute_network.marmot.id ip_cidr_range = "10.0.0.0/24" } # Reserved range that Cloud SQL peers into for private connectivity. resource "google_compute_global_address" "marmot_psa" { name = "marmot-psa-range" purpose = "VPC_PEERING" address_type = "INTERNAL" prefix_length = 16 network = google_compute_network.marmot.id } resource "google_service_networking_connection" "marmot" { network = google_compute_network.marmot.id service = "servicenetworking.googleapis.com" reserved_peering_ranges = [google_compute_global_address.marmot_psa.name] } ``` --- ## The Cloud SQL database A small Postgres instance with a single database and user. ```hcl # database.tf resource "google_sql_database_instance" "marmot" { name = "marmot" database_version = "POSTGRES_16" region = "europe-west1" # Set to true once you're past experimenting. deletion_protection = false settings { edition = "ENTERPRISE" tier = "db-f1-micro" ip_configuration { ipv4_enabled = false private_network = google_compute_network.marmot.id ssl_mode = "ENCRYPTED_ONLY" } } depends_on = [ google_project_service.marmot_apis, google_service_networking_connection.marmot, ] } resource "google_sql_database" "marmot" { name = "marmot" instance = google_sql_database_instance.marmot.name } # Read the password back from Secret Manager, ephemeral (never in state). ephemeral "google_secret_manager_secret_version" "marmot_db_password" { secret = google_secret_manager_secret.marmot_db_password.id version = "latest" depends_on = [google_secret_manager_secret_version.marmot_db_password] } resource "google_sql_user" "marmot" { name = "marmot" instance = google_sql_database_instance.marmot.name password_wo = ephemeral.google_secret_manager_secret_version.marmot_db_password.secret_data password_wo_version = local.marmot_db_password_version } ``` --- ## Secrets Both secrets are generated by ephemeral resources and passed to Cloud Run through Secret Manager without ever landing in Terraform state. Their values are written to the secret versions with the write-only `secret_data_wo` argument, so neither the database password nor the encryption key is stored in the plan or state. ```hcl # secrets.tf locals { marmot_db_password_version = 1 # bump to rotate marmot_encryption_key_version = 1 # locked by ignore_changes } ephemeral "random_password" "marmot_db" { length = 32 special = false } resource "google_secret_manager_secret" "marmot_db_password" { secret_id = "marmot-db-password" replication { user_managed { replicas { location = "europe-west1" } } } } resource "google_secret_manager_secret_version" "marmot_db_password" { secret = google_secret_manager_secret.marmot_db_password.id secret_data_wo = ephemeral.random_password.marmot_db.result secret_data_wo_version = local.marmot_db_password_version } # The encryption key must be base64(32 random bytes) to match # `marmot generate-encryption-key`, which is exactly what .base64 gives us. ephemeral "random_bytes" "marmot_encryption_key" { length = 32 } resource "google_secret_manager_secret" "marmot_encryption_key" { secret_id = "marmot-encryption-key" replication { user_managed { replicas { location = "europe-west1" } } } lifecycle { prevent_destroy = true } } resource "google_secret_manager_secret_version" "marmot_encryption_key" { secret = google_secret_manager_secret.marmot_encryption_key.id secret_data_wo = ephemeral.random_bytes.marmot_encryption_key.base64 secret_data_wo_version = local.marmot_encryption_key_version # Rotating the key breaks every existing encrypted credential, # so a version bump is ignored. lifecycle { ignore_changes = [secret_data_wo_version] } } ``` Secret rotation is straightforward. The `_wo` values are write-only, so Terraform never stores them and only re-reads them when the matching `_wo_version` changes. Since the SQL user reads its password from the secret, bumping `local.marmot_db_password_version` rotates the database and Cloud Run together from one source of truth. The encryption key is the exception. A new key makes every already-encrypted credential unreadable, so we lock it down: `prevent_destroy` keeps a `terraform destroy` from deleting it, and `ignore_changes` on its version makes a bumped local a no-op. Rotating it is then a deliberate two-step (remove the `ignore_changes` block, then bump) rather than something a routine apply can do by accident. See [Terraform, Google Cloud, and secrets](https://www.bschaatsbergen.com/terraform-google-cloud-and-secrets) for more. :::tip The key lives only in Secret Manager. It never touches Terraform state, so Terraform can't regenerate it, and **losing the secret means losing every credential encrypted with it.** `prevent_destroy` guards against an accidental delete, but for real safety read the value once with `gcloud secrets versions access latest --secret=marmot-encryption-key` and keep an offline copy. ::: --- ## Service account and permissions We create a `marmot` service account for the Cloud Run service to run as, and give it read access to the two secrets so the container can pull the database password and encryption key at startup. ```hcl # iam.tf resource "google_service_account" "marmot" { account_id = "marmot" display_name = "Marmot Cloud Run service" } resource "google_secret_manager_secret_iam_member" "marmot_encryption_key" { secret_id = google_secret_manager_secret.marmot_encryption_key.id role = "roles/secretmanager.secretAccessor" member = "serviceAccount:${google_service_account.marmot.email}" } resource "google_secret_manager_secret_iam_member" "marmot_db_password" { secret_id = google_secret_manager_secret.marmot_db_password.id role = "roles/secretmanager.secretAccessor" member = "serviceAccount:${google_service_account.marmot.email}" } ``` These permissions only get Marmot running. The plugins you enable authenticate as this same service account through Application Default Credentials, so when you enable one for GCS or BigQuery, give this service account (or a service account it impersonates) the IAM permissions that plugin needs. --- ## Mirroring the container image Cloud Run can only pull from Artifact Registry, Container Registry, or Docker Hub. Marmot is published to GitHub Container Registry (`ghcr.io`), which Cloud Run won't pull from directly, so we add an Artifact Registry [remote repository](https://cloud.google.com/artifact-registry/docs/repositories/remote-repo) that proxies and caches it. ```hcl # registry.tf resource "google_artifact_registry_repository" "marmot_ghcr" { location = "europe-west1" repository_id = "ghcr-remote" format = "DOCKER" mode = "REMOTE_REPOSITORY" remote_repository_config { description = "Remote mirror of ghcr.io" common_repository { uri = "https://ghcr.io" } } depends_on = [google_project_service.marmot_apis] } ``` Because the repository lives in the same project, Cloud Run's service agent can pull through it without any extra IAM. --- ## The Cloud Run service With everything else in place, here's the service itself. The `vpc_access` block gives the service Direct VPC egress so it can reach the database on its private IP, and the database password and encryption key come in as secret references. ```hcl # run.tf resource "google_cloud_run_v2_service" "marmot" { name = "marmot" location = "europe-west1" ingress = "INGRESS_TRAFFIC_ALL" deletion_protection = false template { service_account = google_service_account.marmot.email scaling { min_instance_count = 0 max_instance_count = 2 } vpc_access { egress = "PRIVATE_RANGES_ONLY" network_interfaces { network = google_compute_network.marmot.id subnetwork = google_compute_subnetwork.marmot.id } } containers { image = "${google_artifact_registry_repository.marmot_ghcr.registry_uri}/marmotdata/marmot:latest" ports { container_port = 8080 } env { name = "MARMOT_DATABASE_HOST" value = google_sql_database_instance.marmot.private_ip_address } env { name = "MARMOT_DATABASE_PORT" value = "5432" } env { name = "MARMOT_DATABASE_USER" value = google_sql_user.marmot.name } env { name = "MARMOT_DATABASE_NAME" value = google_sql_database.marmot.name } env { name = "MARMOT_DATABASE_SSLMODE" value = "require" } env { name = "MARMOT_DATABASE_PASSWORD" value_source { secret_key_ref { secret = google_secret_manager_secret.marmot_db_password.secret_id version = "latest" } } } env { name = "MARMOT_SERVER_ENCRYPTION_KEY" value_source { secret_key_ref { secret = google_secret_manager_secret.marmot_encryption_key.secret_id version = "latest" } } } } } depends_on = [ google_secret_manager_secret_iam_member.marmot_db_password, google_secret_manager_secret_iam_member.marmot_encryption_key, google_sql_user.marmot, ] } ``` Finally, make the service reachable. For a quick start we allow public access. However, for a production deployment lock this down with [authentication](/docs/Configure/Authentication) or IAM before putting real data behind it. ```hcl resource "google_cloud_run_v2_service_iam_member" "marmot_public" { name = google_cloud_run_v2_service.marmot.name location = google_cloud_run_v2_service.marmot.location role = "roles/run.invoker" member = "allUsers" } output "marmot_url" { value = google_cloud_run_v2_service.marmot.uri } ``` --- ## Deploy ```bash terraform init terraform apply ``` Terraform prints the service URL when it finishes: ``` marmot_url = "https://marmot-abc123-ew.a.run.app" ``` Open it, and log in with the default **admin / admin** credentials. The first request may be a little slow. With `min_instance_count = 0` the service scales to zero, so Cloud Run cold-starts the container and runs migrations against the fresh database. Set the minimum to `1` if you'd rather keep one instance warm. --- ## Where to go next That's a complete, self-contained Marmot install. A few things worth doing before it's truly production-ready: - **Set `deletion_protection = true`** on the Cloud SQL instance and the Cloud Run service. - **Add a custom domain and `MARMOT_SERVER_ROOT_URL`**, which is required once you enable OIDC login. - **Restrict ingress** and put [authentication](/docs/Configure/Authentication) in front of the catalog. - **Start ingesting**: point a [plugin](/docs/Plugins) at your warehouses, object storage and message queues. --- ## Connect Claude Desktop to Marmot Most people at your company already have Claude, ChatGPT, Gemini or Cursor installed. These assistants know a lot, but not your organization: where revenue actually lives, which service collects customer analytics, who owns which database, what the schema behind a dashboard looks like. Marmot exposes all of that context through a single MCP server, so the assistants people already use can pull in your organization's metadata. The questions that used to land in a team's Slack channel get answered directly by the person or agent who had them, which is a huge autonomy boost. This post wires up Claude Desktop, end to end. ## Why Claude Desktop Most questions about data come from people who didn't catalog it. Which table holds subscription revenue, what "active user" actually means in the dashboard they're quoting, who owns the pipeline they're about to build on. In practice those questions land in a Slack channel and sit there until someone from the data team has a minute. With Marmot connected to Claude Desktop, people can just ask. "What data do we have on orders?" or "who owns the payments topic?" gets answered from the catalog, with the actual schema and the actual owner. The person asking keeps working instead of waiting, and the data team stops being a routing layer for questions the catalog already answers. It's also another return on the cataloging work. The same descriptions, owners and glossary terms people browse in the Marmot UI now also answer questions in Claude, right in the middle of whatever someone is working on. They ask mid-task, get the answer, and carry on. --- ## Prerequisites You need two things: - A running Marmot instance. If you haven't deployed one yet, follow the [deployment docs](/docs/Deploy/); there are guides for Docker, Kubernetes via the official Helm chart, and more. - Claude Desktop, available from [claude.com/download](https://claude.com/download). You'll also need a Marmot API key for Claude to authenticate with. In Marmot, go to your **Profile**, then **API Keys**, and generate a new key. Claude gets the same permissions as your user account, so all role-based access controls still apply. --- ## Configure Claude Desktop Open Claude Desktop and go to **Settings** → **Developer**: {/* TODO: add screenshots: Claude Desktop Settings -> Developer pane */} Click **Edit Config**. This opens `claude_desktop_config.json`, which lives at: - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` - Windows: `%APPDATA%\Claude\claude_desktop_config.json` - Linux: `~/.config/Claude/claude_desktop_config.json` Add Marmot under `mcpServers`. Claude Desktop speaks to local MCP servers, so we use [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) to bridge to Marmot's built-in MCP endpoint: ```json { ... "mcpServers": { "marmot": { "command": "npx", "args": [ "-y", "mcp-remote", "http://localhost:8080/api/v1/mcp", "--header", "X-API-Key:", "--allow-http" ] } }, ... } ``` Replace `` with the API key you generated earlier, and the URL with wherever your Marmot instance runs. The endpoint is always `/api/v1/mcp` on your Marmot host. :::note Only add `--allow-http` if your Marmot host uses plain HTTP, like the `localhost` example above. If your instance is served over HTTPS, use the `https://` URL and drop the flag. ::: Save the file and restart Claude Desktop. Marmot now shows up as a connected server under Settings → Developer, and Claude has the catalog's tools available: data discovery, ownership lookups, lineage tracing and glossary definitions. --- ## Make sure there's something to find Claude can only surface what's in the catalog, so if your Marmot instance is still empty, populate it first. The easiest way is straight from the Marmot UI: go to Runs, click Create Pipeline and pick the plugin for a source you already have, whether that's PostgreSQL, BigQuery, Kafka, S3 or anything else. The pipeline discovers and catalogs your assets automatically. The [UI guide](/docs/Populating/UI) walks through each step. If you'd rather do it as code, the [Populating docs](/docs/Populating/) cover the CLI, Terraform, Pulumi and the REST API. One source is plenty to follow along with the rest of this post. --- ## Ask your catalog anything Now you can start asking questions in plain language. Here I asked what data we have available on our customers, and which database I'd need if I wanted to build a dashboard on those analytics: {/* TODO: add screenshots: Claude Desktop answering a discovery question via Marmot */} Lineage works the same way. Here I asked which upstream service inserts the data into the warehouse, and where that service is hosted: {/* TODO: add screenshots: Claude Desktop resolving ownership/lineage via Marmot */} This is what a context layer like Marmot buys you. All of that knowledge already existed somewhere in the company, scattered across tools, wikis and people's heads. Asking around on Slack and digging through docs carried us for years, but it's slow and it depends on the right person having time. With a context layer you get the same answers on your own, in seconds. --- ## Where to go next - Descriptions, owners and [glossary](/docs/glossary) terms are what Claude actually answers with, so fill those in for the assets people ask about most. - Everyone on the team connects with their own API key. Permissions follow the user, so nobody sees more through Claude than they would in Marmot itself. - The same endpoint works from [Claude Code](/docs/MCP/claude-code), [Cursor](/docs/MCP/cursor) and any other MCP client; the [MCP docs](/docs/MCP/) have the config for each. --- ## Catalog your Kubernetes clusters A data catalog usually stops at the data. It knows the table that holds subscription revenue and the owner of the payments topic, but not the service that writes to that table or the cluster that service runs in. For a lot of teams the runtime layer is Kubernetes, and it almost never makes it into the catalog. We added three plugins to close that gap: one for self-managed clusters, and one each for the managed offerings we see most, Amazon Elastic Kubernetes Service and Google Kubernetes Engine (_with Azure Kubernetes Service on the way_). They share a single discovery engine, so a namespace, service, its deployment and its cron jobs land in the catalog as assets right next to your databases and topics. Once they are in the same graph you can draw lineage between them, and a table can trace back to the deployment that fills it and the cluster that deployment runs in. This post is on how the Kubernetes plugins work and how to wire one up. ## Why catalog Kubernetes Most of what you want to know about a data asset is really a question about the thing that runs it. Which service writes this table. What does this cron job populate. Is the workload behind this pipeline healthy, and when did it last run. Today those answers live in the cluster, and getting them means finding someone with `kubectl` while everyone else waits. Cataloging the cluster moves those answers into the open. The service, the deployment behind it and its cron jobs land next to your databases and topics, so the runtime and the data it serves sit in one place that anyone, and any agent, can query. You stop routing questions through the person who happens to have cluster access, and the people who own the data can finally see what produces and consumes it. The nice part is how little it costs to get there. Kubernetes does not go stale: each Marmot discovery run reads the current state, not a wiki page that was wrong the moment someone renamed a deployment. And the cluster is already annotated. The labels, owner references, cron schedules and service accounts teams set to operate it come across as metadata for free, with no documentation to write. The plugin only ever reads, so the RBAC it needs is `get` and `list`. --- ## What gets discovered The plugin discovers namespaces, services, deployments, stateful sets and cron jobs, and optionally pods. It links them the same way Kubernetes does internally: a service to the workloads its selector matches, a workload to its pods by owner reference, everything up to its namespace. Cron jobs come with run history built from their recent job runs, so the catalog shows whether last night's job actually succeeded. Pods are off by default; they are short-lived and would churn the catalog constantly, so you opt into `discover_pods` when pod-level visibility is worth it. The same reasoning keeps one-off jobs out: only jobs owned by a cron job are kept, and only as run history. Each asset carries the metadata you would otherwise go digging for: images, replica counts, ports, schedules, service accounts, and so on. The [Kubernetes plugin docs](/docs/Plugins/Kubernetes) list every resource, option and field. --- ## Self-managed, Amazon Elastic Kubernetes Service and Google Kubernetes Engine There are three separate plugins, one per environment. Kubernetes is Kubernetes once you are talking to the API server, so they share a single discovery engine and produce identical assets, lineage and run history. What differs is how each one gets a token to talk to the cluster. - The [Kubernetes plugin](/docs/Plugins/Kubernetes) is for self-managed and on-prem clusters. It uses an in-cluster service account, your kubeconfig, or a host, token and CA you hand it directly. - The [Amazon Elastic Kubernetes Service plugin](/docs/Plugins/EKS) wraps that engine with AWS IAM. You give it a cluster name and region; it looks up the endpoint from the Amazon Elastic Kubernetes Service API and mints a short-lived token from whatever AWS credentials Marmot is running with. - The [Google Kubernetes Engine plugin](/docs/Plugins/GKE) does the same with Google Cloud IAM and an OAuth token. The property worth pointing out: on Amazon Elastic Kubernetes Service and Google Kubernetes Engine there is no static credential to store or rotate. Run Marmot on an instance in the same account or project and it authenticates as the identity it already has, on every run. --- ## Setting it up For any cluster you plan to keep cataloged, use the [Terraform provider](https://registry.terraform.io/providers/marmotdata/marmot/latest/docs). A pipeline is a `marmot_pipeline` resource, so it goes through code review and lives in version control next to the infrastructure it describes. Here is a Google Kubernetes Engine cluster cataloged hourly: ```hcl resource "marmot_pipeline" "prod_gke" { name = "prod-gke" plugin_id = "gke" config = jsonencode({ project_id = "acme-prod" location = "us-central1" cluster = "prod" }) cron_expression = "0 * * * *" # hourly } ``` The [Terraform walkthrough](/blog/configure-marmot-with-terraform) goes deeper on managing pipelines declaratively, and the [Populating docs](/docs/Populating/) cover the CLI, Pulumi and the REST API. --- ## Tying the cluster to the rest of the catalog A cluster on its own is a map of what runs. It gets interesting when it sits next to everything else in the catalog. Your databases, topics and buckets are already there from their own plugins. Your services and cron jobs are now there too. Draw lineage between them and the graph closes. Take the payments API below: the Kubernetes service and its deployment are assets, and so is the MySQL database they write to. The edge between them connects the running workload to the data it produces. Now you can walk the graph either way: from the MySQL table up to the service that writes it and on to the namespace, cluster and cloud it runs in, or back down. Before you drop a column, you can see the deployment that depends on it. An on-call question like "what writes this table, and is that service healthy" becomes a path through the graph instead of a thread across three teams. Marmot serves all of this over [MCP](/docs/MCP/), so the same graph is available to whatever assistant you already use. Point [Claude Desktop](/blog/connect-marmot-to-claude-desktop) at it and "which service writes to the payments database, and is it healthy" gets answered from the catalog, no `kubectl` required. These plugins are experimental for now. If you run them, I want to hear where the discovery or the metadata falls short and what you would want the catalog to show. The fastest way to reach us is Discord. --- ## Migrate from OpenMetadata to Marmot in five minutes This one is for everyone running OpenMetadata and curious about Marmot. The new [OpenMetadata plugin](https://plugins.marmotdata.io/marmotdata/openmetadata) imports your entire instance in one run, about five minutes of setup, and keeps syncing from OpenMetadata until the day you switch it off. Marmot does all of it for free. Don't take our word for it, see for yourself: We built this plugin so you can try Marmot on your own catalog instead of a demo dataset. Everything you spent years curating in OpenMetadata, the descriptions, the owners, the glossary the business argued over, the lineage, is extracted in seconds and yours to play with in Marmot. If Marmot is not for you, turn it off and you have lost nothing. This post is the migration, start to finish. ## One run, your whole catalog Point the plugin at your OpenMetadata host and one run imports tables, topics, buckets, dashboards, pipelines, ML models, API endpoints, the business glossary, lineage and recent pipeline executions. Every entity lands as the technology it actually describes: a table under a Postgres service becomes a PostgreSQL asset, addressed exactly as Marmot's own PostgreSQL plugin would address it. Technologies Marmot has no plugin for yet, such as Snowflake or Looker, come across under their own provider name. The result looks like a catalog Marmot built itself. Descriptions, columns, tags, owners and domains come across on each asset, and nothing loses its trail. Every asset gets an OpenMetadata link that jumps straight to the entity it was imported from, and an `openmetadata` metadata object carrying the fully qualified name, the service and when the entity last changed, so mid-migration any asset in Marmot traces back to its source in one click. The glossary stays a first-class glossary rather than being flattened into tags, and lineage keeps the pipeline that moved the data on each edge. The [plugin docs](https://plugins.marmotdata.io/marmotdata/openmetadata) hold the exact entity-by-entity mapping. --- ## The migration If you do not have Marmot running yet, the [quick start](/docs/quick-start) gets you there with Docker Compose in a couple of minutes. ### 1. Schedule the import Grab a token in OpenMetadata under **Settings**, then **Bots**, and set the import up as a recurring pipeline, for example with the [Terraform provider](https://registry.terraform.io/providers/marmotdata/marmot/latest/docs): ```hcl resource "marmot_pipeline" "openmetadata" { name = "openmetadata-import" plugin_id = "openmetadata" config = jsonencode({ host = "https://openmetadata.company.com" jwt_token = # inject securely }) cron_expression = "0 * * * *" # hourly } ``` For injecting the token securely, use [ephemeral values and resources](https://www.hashicorp.com/en/blog/ephemeral-values-in-terraform), so the token never lands in your state or plan files. Everything is imported by default; the [configuration reference](https://plugins.marmotdata.io/marmotdata/openmetadata) covers scoping down to specific services, and the same config works from the UI wizard, CLI, Pulumi and the REST API. ### 2. Keep working in both catalogs Each scheduled run brings across whatever changed in OpenMetadata, so the two stay in step for as long as the move takes. Re-running is safe, and anything written in Marmot stays as you left it: an edited description is stored separately from the imported one, so the next run refreshes the imported side without touching the edit. The same holds for tags, owners and glossary terms added in Marmot. Curation never pauses for the migration. ### 3. Adopt native plugins one system at a time When you are ready to catalog a system directly, add its own pipeline, for example the [PostgreSQL plugin](https://plugins.marmotdata.io/marmotdata/postgresql) against the database OpenMetadata was describing. Imported assets carry the same identity the native plugin uses, so the native run takes over the existing assets instead of creating a second copy. Nothing gets deleted or re-pointed, and the descriptions people wrote stay put. Work through your systems at whatever pace suits. ### 4. Switch OpenMetadata off When nothing depends on it anymore, stop scheduling the run. The imported assets stay exactly as they are; there is no cliff on the day you turn it off. One warning for that last day: retire the plugin by removing its schedule, not with `marmot ingest --destroy`. Destroy deletes every asset the pipeline ever created, including ones a native pipeline has since taken over. --- ## What you end up with The whole catalog in one graph, whichever system each piece came from. Here is the path OpenMetadata held, a PostgreSQL table and a Kafka topic feeding a Snowflake mart feeding a Looker dashboard, rendered in Marmot: And the glossary your business spent months agreeing on, with definitions, synonyms and assignments intact: All of it runs on one Go binary and a Postgres database, browsable in the UI and served to Claude, Cursor or any other assistant over [MCP](/docs/MCP/). If you are still weighing the move itself, the [full comparison](/resources/marmot-vs-openmetadata) covers footprint, MCP and connectors line by line. The plugin is experimental for now. If you run it against a real OpenMetadata instance, we want to hear what came across wrong and what got skipped that should not have been. The fastest way to reach us is Discord.