Kindly fill up the following to try out our sandbox experience. We will get back to you at the earliest.
Unity Catalog governs everything inside Databricks and stops at its edge. What it covers, the documented limits on lineage and quality, and when it is genuinely enough.
Updated on
Unity Catalog covers Databricks completely and stops at the edge of Databricks, which is the design rather than a criticism of it. Databricks documents that when Unity Catalog is enabled for a workspace it operates beneath every data and AI interaction in that workspace automatically, enforcing access control when someone queries a table or calls a model. Everything it governs has to be registered in a Unity Catalog metastore first, and every policy it enforces is enforced by Databricks compute. Both halves of that sentence are load bearing.
So the useful question is whether your estate fits inside the perimeter it draws, rather than whether Unity Catalog is any good. If it does, you already have your catalog and you should not spend money on a second one. If it does not, the gap is not a feature you can wait for, because it is where the product deliberately ends. The rest of this article sets out both halves in detail, with the Databricks documentation page behind every claim, and finishes with six conditions you can check against your own estate.
Unity Catalog organizes data and AI assets in a three level namespace written as catalog.schema.object. The documentation lists tables, views, volumes, functions, models and services, which now includes model services and MCP services, alongside storage credentials, external locations, connections and shares that sit directly under the metastore. Objects are either managed, where Unity Catalog handles both governance and the underlying file storage lifecycle, or external, where it handles governance only.
That list is wider than most people assume, and it is the strongest thing about the product. A machine learning model, a registered function and a folder of unstructured files are governed by the same privilege model as a table, in the same place. Very few catalogs cover AI assets and tabular data under one grant system without a separate module and a separate purchase.
Access control runs on standard grants plus attribute based access control. In the ABAC model, access is decided by evaluating attributes attached to securable objects, expressed as governed tags, which are defined at the account level with their own access controls over who may create, assign and manage them. A policy is attached at a level in the hierarchy such as a catalog, schema or table and is evaluated dynamically, so a single policy can enforce the same rule across an entire catalog.
Tags themselves are ordinary metadata with documented ceilings: a maximum of 50 tags on a single table or column, at most 1,000 column tags across a whole table, and 256 characters for a tag key or value. The Databricks page carries a warning worth reading before you design a tagging scheme, because it changes what you are allowed to put in a tag name.
Tag data is stored as plain text and may be replicated globally. Do not use tag names, values, or descriptors that could compromise the security of your resources.
Databricks has been building the layer catalogs are usually bought for. Unity Catalog semantics is documented as a collection of tools for defining standard business metrics, terms and organizational structures on top of catalog data, so that people and AI tools read it the same way. It has four parts. Metric views are reusable SQL objects that define and govern business KPIs and are themselves Unity Catalog securable objects. Domains group assets by business purpose. Pages are governed definitions of a business concept such as a critical term, entity or acronym. Certification and deprecation mark an asset as trusted or outdated.
Read the maturity labels before you plan around this. On the pages read on 6 September 2026, domains and subdomains are marked Public Preview and Pages are marked Beta, with account admins controlling access to Pages from the previews page in the account console. Metric views carry no preview label. A glossary in Beta is a real glossary and it is also a glossary whose behavior can change, which matters if a regulator is going to ask you for the definition of a term as it stood eighteen months ago.
Unity Catalog now includes data quality monitoring. Anomaly detection evaluates the completeness and freshness of tables in a schema by analyzing historical patterns. Freshness means how recently a table has been updated, worked out from commit history and compared against when the next update was predicted. Completeness means the number of rows expected to be written to the table in the last 24 hours, compared against a range predicted from the historical row count. It runs on serverless compute, is billed as serverless DBUs under the DATA_QUALITY_MONITORING billing product, and can be enabled on up to 50 schemas at a time at the catalog level.
This is genuinely native quality monitoring and any article claiming Databricks has none is out of date. It is also, by its own documentation, two signals rather than a test suite, which is the point the next section returns to.
| Coverage area | What the Databricks documentation says it does | Where it stops |
|---|---|---|
| Asset governance | Tables, views, volumes, functions, models and services under one three level namespace, with managed and external object types. | Only assets registered in a Unity Catalog metastore. Nothing that is not registered is governed. |
| Access control | Grants plus ABAC policies on governed tags, evaluated dynamically at catalog, schema or table level. | Enforced at the Databricks compute layer, and unreadable by standard or dedicated compute on a runtime earlier than Databricks Runtime 16.4. |
| Lineage | Automatic table and column level lineage captured at runtime from queries run on Databricks, plus model, job and dashboard lineage. | Not preserved across renames, nothing before 1 September 2024, and a rolling 1 year window in the system tables. |
| Business semantics | Metric views, domains, Pages and certification, so people and AI tools read a metric the same way. | Domains are Public Preview and Pages are Beta as documented on 6 September 2026. |
| Data quality | Anomaly detection on freshness and completeness across a schema, on serverless compute. | Two signals, and it does not support views or foreign tables. |
| Assets outside Databricks | Lakehouse Federation for querying external systems, and external metadata objects for declaring lineage into them. | Federation is read only, and external lineage is declared rather than captured. |
This is the part of the topic that is genuinely confusing, and it is worth being exact about, because two different things share one name.
Unity Catalog is a real open source project. It is licensed Apache 2.0, which the LICENSE file in the unitycatalog/unitycatalog repository confirms, and it is a sandbox project with the LF AI and Data Foundation, part of the Linux Foundation. Its README describes an OpenAPI specification and an open source implementation that is compatible with the Apache Hive metastore API and the Apache Iceberg REST catalog API, supporting Delta Lake, Apache Iceberg and Apache Hudi through UniForm, plus Parquet, JSON and CSV, across tables, files, functions and AI models. Its most recent release on 6 September 2026 was 0.6.0, published on 20 August 2026.
Now read the roadmap file in the same repository, because that is where the boundary is drawn in the project's own words. Under Access Control and Governance, basic grants, ownership changes and temporary credential vending for tables, volumes and models are marked as shipped. RBAC, row level filters, column masks, ABAC and lineage are carried with a question mark rather than a release. The 0.6 and later priorities listed at the top of that file are view support, fuller Iceberg REST catalog support, S3 compatible storage and production hardening.
So the lineage graph in Catalog Explorer, the ABAC policies, the row filters and column masks, the semantics layer and the quality monitoring are the managed Databricks service. They are not what you get by running the open source server. If someone tells you Unity Catalog is open source and therefore portable, they are right about the metastore API and the table format interoperability, which is a real and useful thing, and wrong about the governance surface, which is where the buying decision actually sits.
The Lineage in Unity Catalog page states the limitation plainly.
Lineage is not preserved for renamed catalogs, schemas, tables, views, or columns.
This is the limitation with the largest practical cost, because renaming is normal. A table renamed during a migration, a column renamed to match a new naming standard, a schema renamed when a team reorganizes: each one severs the recorded history at that point. The graph looks healthy afterward and simply starts again. Nothing warns you, and the loss is only visible when someone asks a question about the period before the rename.
If you are about to run a naming standardization exercise, that is the moment to export what the graph currently holds, because it will not be there afterward. This is also the clearest argument for keeping lineage in a layer that is governed independently of the objects it describes, which is the design behind approval gated lineage that is changed through a controlled flow rather than silently.
Most articles quote one retention number for Unity Catalog lineage. There are two, and they disagree. Lineage shown in Catalog Explorer is retained indefinitely. The lineage system tables, system.access.table_lineage and system.access.column_lineage, retain a rolling 1 year window, and events older than a year are removed on a rolling basis.
That distinction decides what you can automate. Anything a human clicks through in Catalog Explorer has the long history. Anything you query in SQL, feed to a compliance report, push into a dashboard or use to answer an auditor programmatically has one year. Below both sits a hard floor: lineage data captured before 1 September 2024 is not available at all, on either path. If your obligation is to show a five year trail, Unity Catalog on its own cannot produce it today, and no configuration change will make it.
Databricks says it captures lineage to the column level as much as possible, and then names the cases where it cannot. The lineage page lists two. Column lineage is not captured when the source or the target is referenced as a path rather than a table name, so a query reading from a storage path directly loses its column mapping. It is also lost through user defined functions, which obscure the mapping between source and target columns.
A third case is documented, but on a different page, and it is the one most often quoted without a source. The lineage system tables reference says that the column lineage table does not include events that have no source, and gives inserting into a column using explicit values as the example. In practice that means a literal INSERT INTO with VALUES produces no column lineage record, which quietly affects reference data, seed tables and manually corrected rows. If you check the main lineage page for that sentence you will not find it, so cite the system tables reference when you repeat it.
The list of things not captured at all is worth reading in full once. Resilient Distributed Datasets, global temp views, tables under system.information_schema, and jobs submitted through the Jobs API runs submit request or the spark submit task type, which still produce table and column lineage but lose the link to the job run. Lineage from Lakeflow pipelines is incomplete where a pipeline uses private tables. And transactions emit lineage as each read and write occurs, so a rolled back transaction leaves its lineage events behind.
Lineage is captured per metastore, so lineage recorded in one workspace is visible in any other workspace sharing that metastore, provided the user has the object permissions. What does not travel is the detail of workspace level objects. Databricks documents that detailed information about workspace level objects such as notebooks and dashboards in other workspaces is masked, and that such detail is visible only in the workspace where the object was created.
For a single team on one workspace this never comes up. For a bank running separate workspaces per business unit against a shared metastore, it means the answer to which notebook wrote this column is available to the team that owns the notebook and masked to everyone else. Objects a user lacks BROWSE or SELECT on appear as masked nodes that cannot be expanded, which is correct behavior for access control and is also the reason a central governance team often sees a less complete graph than the engineers do.
Anomaly detection is real, native and useful, and it monitors freshness and completeness. It does not run business rules. There is no accepted value list, no cross table reconciliation, no referential check, no column level assertion that a currency code is one of a known set. Databricks is explicit that anomaly detection does not support views or foreign tables, and that its completeness judgment does not take into account metrics such as the fraction of nulls, zero values or NaN. Percent null tracking and completeness slicing by column value are documented as Beta features, and the core capability is documented as Public Preview.
So a table can arrive on time, with exactly the expected number of rows, entirely full of wrong values, and pass. Freshness and completeness are the two signals the feature claims to provide, so that outcome is correct behavior rather than a defect. It does mean that if your quality problem is correctness rather than arrival, this is not the layer that catches it, which is the distinction we draw between data quality testing and data observability.
Databricks is candid about this. Unity Catalog automatically captures runtime lineage for queries run on Databricks, and for workloads that run elsewhere, such as first mile ETL or last mile BI, it lets you add external lineage metadata to augment what it captured. You do that manually through the UI, the APIs or the Python SDK, or automatically through Lakeflow Connect managed ingestion pipelines. You create an external metadata object, give it a system type and an entity type, optionally map its columns, and then declare the relationship. Creating one requires the CREATE EXTERNAL METADATA privilege on the metastore.
Three documented facts decide how far this carries. External lineage is not recorded in the lineage system tables, so it is invisible to exactly the SQL path you would use to automate a report. You can create up to 10,000 external metadata objects and 100,000 external lineage relationships per metastore. And the relationship is only as current as the last time somebody declared it, because outside Databricks there is no query to observe. That is the difference between a graph that is captured and a graph that is maintained, and it is the same difference we set out in our guide to how automated lineage is actually collected.
Lakehouse Federation is the other route across the boundary, and it is read only. Query federation covers MySQL, PostgreSQL, Teradata, Oracle, Amazon Redshift, Salesforce Data 360, Snowflake, Microsoft SQL Server, Azure Synapse, Google BigQuery and Databricks itself. Catalog federation covers the legacy Databricks Hive metastore, an external Hive metastore, AWS Glue, Salesforce Data 360, Snowflake and Palantir Foundry. Databricks recommends Lakeflow Connect instead where performance on higher data volumes and lower latency matter. Federation lets you query and govern the read path into those systems. It does not govern what happens inside them.
Databricks states that you can create one metastore per region and attach it to any number of workspaces in that region, and that you must have one metastore for each region in which your organization operates. Users have to be on a workspace attached to a metastore in their region to work with Unity Catalog at all.
For a business in one region this is a non issue. For a group operating in Singapore, Sydney and Jakarta it is an architectural fact with governance consequences: three metastores, three sets of grants, three lineage graphs, and no single view of the estate inside Unity Catalog. Anyone who needs one pane across all three has to build it or buy it. This is the constraint most likely to be discovered late, because it does not appear until the second region goes live.
Here is the test. Every one of these six has to be true. If they are, you already have your catalog, and a second one adds cost and a second place for the same metadata to go stale.
A team running a single region, Databricks only analytics estate, with a stable naming scheme and a quality problem that is mostly about pipelines arriving on time, should not buy a data catalog. They should turn on ABAC, adopt governed tags properly, enable anomaly detection on the schemas that matter and spend the budget elsewhere. Anyone who tells that team otherwise is selling.
The conditions above fail in a small number of recognizable ways, and they tend to fail together.
The most common is a lakehouse that is the center of the estate rather than the whole of it. Data arrives from an operational database nobody is allowed to connect directly, leaves into a BI layer that then extracts it, and is copied into a system of record that never touches Databricks. Unity Catalog governs the middle of that journey precisely and sees the ends only as far as somebody declared them. The same shape appears when a regulator asks about a period longer than the retention windows hold, when a group operates in more than one region and has to answer one question across all of them, or when the quality obligation is a contract between a producing team and a consuming team rather than an alert about a late table.
The honest general case is worth naming, because this shape is common to the whole category rather than unique to Databricks. A governance layer built into a data platform is excellent inside that platform and stops at its edge, and the same sentence holds for the equivalent layer in Snowflake. Neither vendor should be expected to fix that, because it describes what a platform native layer is for.
If your estate fits inside the six conditions, stop reading here and keep your money. This section is for the case where it does not.
Decube is a data trust platform where catalog, lineage, quality and observability are built as one system rather than assembled from separate purchases. The two things we consider genuine differentiators are both governance decisions rather than feature counts. The first is approval gated lineage: lineage changes go through a structured approval flow, so the graph is governed the way the data is, which is the direct answer to a history that can be severed by a rename with nothing recorded. The second is dynamic thresholding on quality tests, across 12 test types with no code and custom SQL options, bulk configuration and alert grouping. Observability is native rather than a partner integration: pipeline health, freshness, volume, schema change detection and machine learning based anomaly detection. Data contracts between producers and consumers are a first class object with SQL based test enforcement.
On governance the model is policy driven tagging and classification, automatic PII classification, role based access, group management and approval workflows, built for regulated financial services where an auditor asks who approved a change and when. Deployment is SaaS and measured in weeks without professional services, and pricing is published rather than quoted: Starter at 175 US dollars per user per month from 21,000 US dollars a year with a minimum of 10 users, and Growth at 225 US dollars per user per month from 54,000 US dollars a year with a minimum of 20 users.
One thing we will not do on this page is publish a connector list we cannot stand behind in detail, because the whole argument above is that coverage claims should be checkable. If the deciding question for you is whether our quality engine monitors a specific system in your estate, ask us for the current source list and check it against your own inventory before you decide anything.
| Platform | What it governs | Native quality testing | Native observability |
|---|---|---|---|
| Decube | Catalog, lineage, quality and observability across connected systems, with policy driven tagging, PII classification and approval workflows. | Yes | Yes |
| Databricks Unity Catalog | Assets registered in a Unity Catalog metastore, with policies enforced at the Databricks compute layer. | Partial | Partial |
Partial there reflects what the documentation supports, and it is not meant as a slight. Databricks ships native freshness and completeness anomaly detection, which is real quality monitoring, and that monitoring does not cover views or foreign tables and does not evaluate business rules. Anyone writing that Databricks has no native quality capability at all is repeating a claim that stopped being true.
For a Databricks only estate it usually does. Unity Catalog governs tables, views, volumes, functions, models and services in a three level namespace, with grants, attribute based access control on governed tags, automatic lineage, a business semantics layer and freshness and completeness anomaly detection. It replaces a separate catalog when every asset you govern is registered in a Unity Catalog metastore and every consumer reads through Databricks compute, because those are the two conditions its governance depends on. It does not replace one when a meaningful part of your estate sits outside Databricks, because Databricks documents that lineage into external systems is declared through external metadata objects rather than captured, and that Lakehouse Federation is read only.
Both things are true of different products that share a name. The open source Unity Catalog project is licensed Apache 2.0, is a sandbox project with the LF AI and Data Foundation, and reached release 0.6.0 on 20 August 2026. It provides an OpenAPI specification and a server compatible with the Apache Hive metastore API and the Apache Iceberg REST catalog API, covering tables, files, functions and AI models, with basic grants and temporary credential vending. Its own roadmap file still carries RBAC, row level filters, column masks, ABAC and lineage as open questions rather than shipped features. The lineage graph, the ABAC policies, the semantics layer and the quality monitoring are part of the managed Databricks service, which requires an account on the Premium plan or above.
Not automatically. Databricks captures runtime lineage for queries that run on Databricks, and for workloads that run elsewhere it lets you add external lineage metadata manually through the UI, the APIs or the Python SDK, or automatically through Lakeflow Connect managed ingestion pipelines. Three documented limits decide how far this carries: external lineage is not recorded in the lineage system tables, you can create up to 10,000 external metadata objects and 100,000 external lineage relationships per metastore, and a declared relationship is only as current as the last time somebody declared it.
There are two different retention windows. Lineage displayed in Catalog Explorer is retained indefinitely. The lineage system tables, system.access.table_lineage and system.access.column_lineage, retain a rolling 1 year window, and events older than a year are removed on a rolling basis. Below both is a hard floor: lineage data captured before 1 September 2024 is not available. So anything a person clicks through has the long history and anything you query in SQL has one year.
It does freshness and completeness monitoring rather than rule based testing. Anomaly detection analyzes historical patterns to predict when a table should have been updated and how many rows should have landed in the last 24 hours, and flags tables that are stale or incomplete. Databricks documents that it does not support views or foreign tables, and that its completeness judgment does not take into account metrics such as the fraction of nulls, zero values or NaN. Percent null tracking and completeness slicing by column value are documented as Beta. A table can therefore arrive on time with the expected row count and still contain wrong values.
No. Databricks documents that you can create one metastore per region and attach it to any number of workspaces in that region, and that you must have one metastore for each region in which your organization operates. Users must be on a workspace attached to a metastore in their region. An organization operating in three regions therefore runs three metastores with three sets of grants and three lineage graphs, and Unity Catalog does not provide a single governed view across them.
First step to AI Readiness
Table of Contents
Sneak peek from the data world.
Bring lineage, quality, governance, and AI together in one platform—so your teams and AI agents can act with confidence.
_For%20light%20backgrounds.svg)
.svg){kind=small}
.webp)