One post tagged with "kubernetes"

A data catalog usually stops at the data. It knows the table that holds subscription revenue and the owner of the payments topic, but not the service that writes to that table or the cluster that service runs in. For a lot of teams the runtime layer is Kubernetes, and it almost never makes it into the catalog.

We added three plugins to close that gap: one for self-managed clusters, and one each for the managed offerings we see most, Amazon Elastic Kubernetes Service and Google Kubernetes Engine (with Azure Kubernetes Service on the way). They share a single discovery engine, so a namespace, service, its deployment and its cron jobs land in the catalog as assets right next to your databases and topics. Once they are in the same graph you can draw lineage between them, and a table can trace back to the deployment that fills it and the cluster that deployment runs in.

This post is on how the Kubernetes plugins work and how to wire one up.