Kindly fill up the following to try out our sandbox experience. We will get back to you at the earliest.
Which layer of your architecture holds data lineage, how it fits with the catalog and warehouse, who owns it, what it costs, and what fails without it.
Updated on
Data lineage is the documented record of where data originates, how it moves through systems, how it transforms, and where it is finally consumed. It answers three questions every data driven enterprise needs to answer: where did this data come from, what happened to it along the way, and what depends on it now.
This page assumes you already know that much and takes the next step. If you want the definitions, the types and the techniques first, the data lineage tracking guide covers them properly. What follows here is the architecture question: where lineage belongs in an enterprise stack, who runs it, what it costs, and what happens when it is treated as a purchase.
Lineage is rarely a box you buy and place on an architecture diagram. Four different layers of your stack can each produce it, at different resolutions and with different blind spots. Deciding which layer holds it is the architecture decision, and almost every problem an enterprise has with lineage traces back to that decision being made by accident.
Each layer already sees part of your pipeline as a side effect of doing its own job. The question is how much of the path it can see, and whether it can see past its own boundary.
| Layer | Examples of what sits there | What its lineage can see | Where it stops |
|---|---|---|---|
| Transformation layer | dbt, Coalesce, SQL procedures | Every model, join and column inside the transformation project itself, usually at column level because it parses the SQL it compiles. | The moment data arrives from outside the project or leaves it for a dashboard. It cannot see the ingestion that fed it or the report that consumes it. |
| Warehouse layer | Snowflake Horizon, Databricks Unity Catalog, BigQuery | Every query run against that warehouse, so any table to table or column to column path inside the platform is captured without configuration. | Anything outside that one platform. A second warehouse, an on premise system, a BI tool or a model training job in another environment is invisible to it. |
| Observability layer | Data quality and pipeline monitoring platforms | The paths between the assets it monitors, which is enough to route an alert to the tables downstream of a failure. | Assets it does not monitor, and the business meaning of the path. It maps what it watches, not what exists. |
| Catalog layer | Decube, and other metadata platforms | Paths that cross tools, because the graph is assembled from every connected system rather than from one of them, and it can carry ownership, classification and business definitions on the same nodes. | Systems it has no connector for, unless those systems can be documented manually or pushed in through an API. |
The pattern in that table is the whole argument. Three of the four layers produce lineage as a by product of running something else, and each one can only see inside its own boundary. The catalog layer is the only one that is not itself a tool in the pipeline, which is exactly why it can see a path that crosses several.
The rule is short. Buy lineage at the layer that matches the widest question you have to answer, not the deepest one.
The trap is that the first two are cheap and arrive by default, so a lot of enterprises conclude they already have lineage. They have three partial graphs that do not join up, and nobody notices until an auditor asks for a path that crosses all three.
Most enterprises treat data lineage as a governance tool, a compliance checkbox for BCBS 239, GDPR or APRA. That framing undersells it badly.
Without lineage, your data catalog is an inventory without context. Your data observability platform can detect an anomaly but cannot tell you which downstream dashboards are now wrong, or which machine learning feature store received the corrupted input. Your governance team can define policies but cannot enforce them at column level, because it cannot see which columns are in scope. Lineage is the layer that turns each of those from a list into an answer.
These two are complementary rather than competing, and teams confuse them constantly. The catalog tells you what a dataset is and who owns it. Lineage tells you how it was made and what depends on it. Together they form the context layer that makes data trustworthy at enterprise scale.
The same confusion happens between five different things an enterprise buys, and it is the reason so many organizations pay twice for overlapping tools. This is the boundary an architect actually needs.
| Component | The question it answers | What it cannot answer alone |
|---|---|---|
| Data catalog | What is this dataset, what does the business call it, who owns it, and what is it classified as? | How it was built, and what breaks if it changes. |
| Data lineage | Where did this field come from, what happened to it, and what depends on it right now? | Whether the data in it is correct today. |
| Data warehouse | Where does the data physically live and how fast can it be queried? | Anything about data that lives in another platform. |
| BI and reporting layer | What is the number, and who is looking at it? | Where the number came from, and whether the source changed this morning. |
| Data observability | Is this data healthy right now, and did anything break overnight? | What the break affects, and who needs to be told. |
| Governance function | What is the policy, who is accountable, and can we prove it to a regulator? | Which specific columns and pipelines the policy applies to. |
Read down the last column and the pattern is hard to miss. Four of the six cannot finish their own job without a lineage graph to point at. That is what people mean when they call lineage foundational, and it is why treating it as a feature of one of the other five is the wrong shape. If you want the catalog side of this in more depth, the data catalog and metadata management guide covers it, and the post on pairing a data catalog with data observability covers where those two meet.
Table level lineage maps which tables feed which other tables. It gives a high level view of data flows across the stack and is useful for understanding pipeline architecture. It is not enough for impact analysis or regulatory reporting, because it cannot tell you which field inside the table was affected.
Column level lineage traces individual fields through every transformation, join and aggregation. It is the resolution required for BCBS 239 work in banking, for GDPR data subject requests, and for validating the features that feed a model. Decube's automated column level lineage maps these dependencies without manual annotation, including across dbt models, Snowflake views and Spark jobs, and there is a fuller write up of what column level lineage changes in practice.
Business lineage translates technical pipeline maps into language business users understand. Instead of naming a table, then a dbt model, then a warehouse view, then a workbook, business lineage says that the customer revenue metric feeds the quarterly board report. This is the layer that makes lineage useful to a chief data officer and a data steward, not only to a data engineer.
Process lineage captures the human steps in data workflows: who approved a transformation, which team owns a pipeline, when a business rule was last changed. It is the type that matters most for audit trails in regulated environments, and the one most often missing.
This is the question the ranking pages do not answer, and it is the one that decides whether a lineage program survives its second year. Lineage has an unusual ownership shape because it is produced by systems that different teams run, consumed by teams that run none of them, and depended on by a governance function that owns no systems at all.
The failure mode is predictable. Lineage gets bought by a governance team, implemented by a platform team, ignored by the domain teams whose systems it needs to reach, and then blamed for being incomplete.
| Model | How it works | Where it works | Where it breaks |
|---|---|---|---|
| Central platform ownership | One data platform team owns the graph, the connectors and the coverage, and domain teams are consumers. | Organizations with one warehouse, a single transformation project and fewer than roughly a dozen source systems. It is the fastest way to get a trustworthy graph standing up. | As soon as the number of source systems grows past what one team can maintain connectors for. The backlog becomes the bottleneck and domains route around it. |
| Federated domain ownership | A central team owns the platform and the standard. Each domain owns the completeness and accuracy of its own assets on the graph. | Large organizations, and any organization that has moved to domain owned data products. It is the only model that scales past a few dozen sources. | When the central standard is not enforced. Without a single definition of what covered means, each domain sets its own bar and the graph stops being comparable across domains. |
| Governance office ownership | The governance or risk function owns lineage as a compliance artefact and commissions the work from engineering. | Heavily regulated firms where the primary use is evidence for a supervisor, and where the governance function holds the budget. | Almost immediately, for engineering use. A graph built to satisfy an auditor is refreshed on an audit cycle, and lineage refreshed quarterly is useless for impact analysis on a Tuesday afternoon. |
Most enterprises above a few hundred people end up in the federated model whether they plan for it or not, because that is where the source systems already sit. If your organization has moved toward domain owned data products, lineage becomes a cross domain coordination problem before it becomes a tooling problem, which is the same shift described in the write up on data mesh architecture.
| Role | What this role decides | What breaks when nobody holds it |
|---|---|---|
| Lineage platform owner | Which systems are connected, how often the graph refreshes, and what the coverage target is. Usually sits in the data platform team. | The graph goes stale without anyone noticing, because staleness has no owner and produces no alert. |
| Domain data owner | Whether the assets in their domain are on the graph, correctly classified, and correctly attributed. | Coverage stalls at whatever the connectors happened to reach. The gaps are always in the oldest and most regulated systems. |
| Governance lead | What has to be traceable, to what resolution, and to satisfy which obligation. | Everything gets mapped at the same resolution, which is both too expensive for the unimportant assets and too shallow for the regulated ones. |
| Executive sponsor | Whether lineage is funded as infrastructure with a running budget or as a project with an end date. | The program is funded to a launch and not to a steady state, so year two has no budget and the graph quietly decays. |
The staffing shape that works is smaller than most people expect. A platform owner who spends part of a week on the graph, a named data owner in each domain who spends a couple of hours a month confirming their own assets, and a governance lead who sets the resolution rule once and revisits it annually. What does not work is a full time team documenting other people's pipelines by hand, which is how lineage was done before it could be captured automatically and why so many organizations still think it is expensive.
Very few pages will give you a number here, so here is one. Decube publishes its pricing rather than hiding it behind a form.
| Plan | Per user per month | Annual minimum | Minimum users | Quality monitors |
|---|---|---|---|---|
| Starter | 175 USD | 21,000 USD | 10 | 1,000 |
| Growth | 225 USD | 54,000 USD | 20 | 3,000 |
| Enterprise | Custom | Custom | Large teams | Unlimited |
Every tier includes automated lineage, the catalog and business glossary, schema drift detection and governance features, along with API access and single sign on. The full breakdown is on the Decube pricing page. Treat that annual minimum as the order of magnitude for a lineage and catalog layer at this end of the market rather than as a quote.
The license is usually the smallest line in the total. These four are the ones that surprise people, and none of them appears in a vendor comparison table.
Everything above is easier in one warehouse. Enterprise lineage is hard because the enterprise spreads across two clouds, a mainframe, a core banking system, four transformation projects owned by four separate teams, and a BI tool that somebody in finance has been extending with spreadsheets since 2018.
A lineage graph is only as complete as its connectors. The moment a path reaches a system the platform cannot connect to, an on premise database, a legacy core system, anything without a native integration, the path stops there. What makes this dangerous is that the graph does not say it stopped. It shows a node with no parent, which looks exactly like a source system, and an auditor asking where a figure came from gets an answer that is confident and wrong.
The fix is to let a system be documented in the catalog without a live connection, so the undocumented upstream node appears on the graph and joins the path, either edited by hand or pushed in on a schedule through an API. The short walkthrough below shows that being done in Decube with a virtual source, which is the mechanism that closes the gap for an on premise or legacy system.
Whichever platform you run, the coverage question is the one to ask in a trial: not how many integrations are on the website, but whether your specific unconnected system can be represented on the graph at all. The sibling article on how automated lineage capture works and where it fails goes through the capture methods in detail and explains how to audit what your graph actually reaches.
The instinct in a large organization is to centralize, and it works right up to the point where the central team becomes the queue everybody waits in. The workable split is to centralize the things that must be identical and to federate the things that must be local.
A useful test for whether the split is right: if a domain team wants a new source system on the graph, can they do it themselves within a sprint? If the answer is that they have to raise a ticket with a central team, the model will not hold as the estate grows. The practices for setting that standard are covered in the companion article on data lineage best practices.
Data incidents are costly. An hour of wrong numbers in a dashboard, a failed prediction or a corrupted feed can trigger downstream decisions of real consequence. Column level lineage converts a search in the dark into a guided trace. Engineers pinpoint the source of a quality issue, see every affected downstream asset, and resolve the incident before business users notice.
Data infrastructure is not static. Teams deploy schema changes, refactor pipelines and migrate cloud systems continuously. Without lineage, every change is a gamble. With it, an engineer can ask what breaks if a particular column is dropped from a particular table and get back a complete list of dependent views, models, dashboards and features. Change management becomes predictable instead of reactive.
Governance frameworks such as DAMA, BCBS 239, SOX and GDPR require demonstrable control over data flows. Manual documentation of those flows is unsustainable at enterprise scale. Automated lineage produces a continuously updated, auditable map of how data moves through the organization, which is what a supervisor actually wants to see. It is also what lets a governance team apply a policy at the right scope, to the specific columns and pipelines that process regulated data rather than to everything. Pairing that with a metadata management platform is how the policy tags stay attached as the data moves.
Lineage reveals how many pipelines are quietly doing the same job. In most large enterprises, several teams have independently built transformation logic that derives the same figure, and nobody can see it because nobody can see across the teams. Lineage makes the overlap visible, which lets a platform team consolidate the redundant pipelines, cut the compute they were burning, and remove the governance risk of two teams computing the same metric two different ways.
A new data engineer joining a team at a large bank or insurer spends weeks learning which tables feed which reports. With lineage visualized in a platform, that same engineer can trace any flow interactively on their first day. The gain compounds across every hire, and it is the benefit that shows up soonest.
A model is only as trustworthy as the data it was trained or grounded on. When a large language model produces a wrong answer from your enterprise data, the first question is where that data came from and what happened to it before the model saw it. Without lineage there is no way to answer that, and without an answer you cannot fix the model, explain the failure to a supervisor, or stop it happening again.
Decube's context layer connects lineage directly to data observability and catalog metadata, so an AI system is reading governed context rather than whatever happened to be indexed.
An agent differs from a dashboard in one way that matters for governance: it chooses what to read. A dashboard queries what an engineer pointed it at. An agent searches, picks an asset and acts on it, which means the governance layer has to be readable by the agent rather than by a person in a meeting.
Four things have to be true of your data before an agent is safe to point at it, and they are the four things a governance platform is for.
A platform that does all four in one place is what most people are describing when they ask for a tool that combines data lineage with data governance. Decube covers all four from a single metadata layer, which is what its data lineage and metadata management products are for, and there is a fuller comparison of the market in the roundup of data governance tools.
Regulation is moving the same way. Under the EU AI Act, obligations for general purpose AI models have applied since 2 August 2025 for models placed on the market from that date, and the European Commission gained enforcement powers on 2 August 2026, per the Commission's own guidance for general purpose AI providers. Being able to show what data an automated system consumed is becoming an obligation rather than a good practice, and lineage is the only artefact that shows it.
Lineage is the evidence layer for regulatory compliance across every major data regulation active in Asia Pacific and globally.
Organizations using Decube combine lineage with automated policy tagging to hold a continuous compliance position across every change event, rather than assembling evidence in the weeks before an audit. The Basel Committee text itself is published at bis.org.
The platform is rarely what separates the programs that work from the ones that quietly die. What separates them is whether the organization treated lineage as something it bought or as something it runs. Here is what the bought kind looks like when it goes wrong.
| Failure | What it looks like from the inside | What prevents it |
|---|---|---|
| Bought, launched, unowned | The graph is accurate on the day of the launch demo and drifts from that afternoon. Six months later people check it, find it wrong once, and stop opening it. | A named platform owner with a refresh target, and an alert when the graph misses it. |
| Coverage measured as a percentage | A dashboard says coverage is high, and it is, because it counts the assets the connectors reached and cannot count the ones they did not. | Define covered as a test on the assets that matter: an unbroken column level path from each regulated report and production model back to a named source system. |
| Table level everywhere | Impact analysis returns a list of tables rather than fields, so every schema change still ends in a manual investigation and nobody uses the graph for change management. | Column level resolution wherever regulation or a model touches the data, from the first day rather than as a later upgrade. |
| Lineage as a picture, not a trigger | The graph is a diagram people look at. It changes nothing automatically, so governance stays a review meeting. | Wire lineage into the workflows: policy applied when classified data appears, owners notified when a change hits a regulated report. |
| Funded as a project | Budget covers the implementation and stops. Year two has no owner time, connectors go stale, and the graph is quietly abandoned. | An executive sponsor who funds it as running infrastructure with an annual line, in the same way the warehouse is funded. |
| One graph per team | Each domain stands up its own lineage inside its own tools. Every graph is correct and none of them can answer a question that crosses two domains. | One shared graph at the catalog layer, with the coverage federated to the domains rather than the platform. |
Before choosing any tooling, map the scope of your stack. Identify your transformation layer, your orchestration layer, your warehouse or warehouses, and your consumption layer. Then write down the systems that none of them touch, because those are the ones that decide the outcome. Tooling that cannot parse your specific stack will produce an incomplete graph, and an incomplete graph that does not say it is incomplete is the worst result available.
Table level lineage is faster to stand up and creates debt. Regulatory obligations and AI use cases both need column level resolution, and retrofitting it later means redoing the work. Build for column level from the start, even if you begin with a subset of high priority domains.
Manual lineage documentation degrades within weeks, because engineering teams move faster than documentation processes. Automated capture through query parsing, pipeline metadata APIs and connector native integration is the only approach that holds. Decube's automated column level lineage captures from connected systems continuously, without engineers annotating flows by hand.
Lineage alone tells you that one table feeds one report. Lineage connected to observability tells you that the table fed the report with data that failed its quality checks yesterday, and that the report is read by the finance leadership team. That combination is what turns data trust from a claim into an operation.
The goal reaches past giving engineers a graph they can query. Data stewards, analysts and chief data officers also need a view of data flows they can act on, which means translating the technical graph into business terms, surfacing it in the catalog interface, and connecting it to the business glossary. Decube's metadata management platform does that translation automatically, so the same graph serves an engineer debugging a pipeline and an executive certifying a report.
Lineage should trigger governance actions. When a new column containing personal data is detected, the policy should be applied automatically. When a schema change is going to hit a regulated report, the data owner should be told before deployment rather than after. Lineage as an active trigger rather than a passive document is the difference between a working program and a compliance performance.
If you are an architect deciding where lineage belongs, the sequence is short. Pick the layer using the rule above, name the platform owner and the domain owners before you sign anything, define covered as a test rather than a percentage, and fund it as a running line rather than a project. If you want to see what a column level graph looks like across a real stack, book a Decube demo, or read the walkthrough of what column level lineage changes in practice first.
An enterprise architect needs four things that fit together rather than four products. The data catalog supplies the inventory and the business meaning. Data lineage supplies the path between assets, so impact analysis and traceability are possible. Data quality supplies the health signal on each asset. Data governance supplies the policy, the accountability and the evidence for a regulator. The architecture requirement is that all four attach to the same metadata layer, because a policy that cannot point at specific columns cannot be enforced, and a quality alert that cannot see downstream cannot be routed. Buying them separately is what produces four partial views that never reconcile.
Enterprise data lineage is a lineage graph that crosses tool and platform boundaries rather than staying inside one of them. Lineage inside a single transformation project or a single warehouse is useful but partial, because it stops at the boundary of the system that produced it. Enterprise lineage is assembled at the catalog layer from every connected system, holds column level resolution where regulation or a model touches the data, carries ownership and classification on the same nodes, and covers systems that have no native connector by letting them be documented or pushed in through an API.
Data lineage tracking is the continuous capture of how data moves and changes, as opposed to a one time documentation exercise. A tracked graph updates itself as pipelines run, so it reflects the stack as it is today rather than as it was on the day someone drew it. Tracking is what makes lineage usable for impact analysis and incident response, because a graph refreshed on an audit cycle is already out of date by the time an engineer needs it.
The platforms that genuinely combine the two are metadata platforms at the catalog layer, because governance needs to attach policy to the same nodes the lineage graph is built from. Decube does this in one layer: automated column level lineage, the catalog and business glossary, classification and policy tagging, and data quality monitoring all read and write the same metadata. Other platforms in this category include Atlan, Collibra, Alation, Informatica and OvalEdge. Warehouse native lineage and transformation tool lineage do not combine with governance in the same way, because neither can see past its own platform boundary, and a policy that only covers one platform is not a policy.
Yes, and on a modern catalog platform that is the point. Once source systems are connected, the lineage graph is captured automatically from queries, transformation project metadata and connector APIs, so a governance team can trace a field from a regulatory report back to its source without asking an engineer to write anything. What a governance team cannot do alone is extend coverage to a system that has no connector, or decide that a legacy platform should be documented as a virtual source. That still needs the platform owner and the domain owner, which is why lineage has a shared ownership model rather than a single one.
By producing an artefact rather than a policy document. The evidence a supervisor accepts is a traceable path: this figure in this report was computed from these columns in these source systems, through these named transformations, on this date, with these classifications and this owner attached at every step. An automated lineage graph produces that on demand. A manually maintained document produces a claim that the reviewer then has to test. The difference is why regulated firms move to automated capture even when their manual documentation is technically complete.
BCBS 239 is the Basel Committee on Banking Supervision publication Principles for effective risk data aggregation and risk reporting, published on 9 January 2013 in response to the finding that many banks, including globally systemic ones, could not aggregate risk exposures fully, quickly and accurately. It does not name a lineage product, but demonstrating that risk data can be traced from source systems to a regulatory submission is what banks are asked to evidence, and column level lineage is the practical way to produce that trace continuously rather than reassembling it before each review.
Know your customer processes depend on customer attributes that are assembled from several source systems, often including one that predates the current stack. Lineage is used in two ways. First, to prove where each attribute in a customer risk assessment came from, which is what a supervisor asks about when a decision is challenged. Second, to run the reverse query when a source changes: if an identity field is corrected or a screening list is updated, lineage returns every assessment, report and model that consumed the old value and now needs rerunning.
Model lineage is the provenance chain for a machine learning model rather than for a table: which datasets and which columns produced each feature, which version of each dataset was used for training, and which downstream decisions the model output fed. Compliance needs it because explaining an automated decision means explaining its inputs. Without model lineage an organization can describe how a model works but cannot show what it was built from, which is the part a supervisor asks for.
Data lineage is the history of a data asset: where it came from, what was done to it, and where it ended up. It is the record that lets an organization answer a regulator, find the cause of a wrong number, and decide whether an output is safe to rely on.
Table level lineage shows which tables feed which other tables, which is useful for an architecture overview and not enough for precise work. Column level lineage traces individual fields through every transformation, join and calculation, so you can see exactly where a metric originated and which downstream reports depend on it. Column level resolution is what regulatory traceability, data subject requests and model feature validation actually require.
They are complementary. Quality monitors detect that something is wrong, such as a spike in null values, a volume anomaly or a freshness failure. Lineage explains what else it affects. A quality alert on its own tells you a table is broken. Lineage tells you that the broken table feeds a fraud detection model and three executive dashboards. The combination is what makes incident response fast and accurate instead of a manual investigation.
In most regulated industries, yes in practice even where no regulation names the word. BCBS 239 in banking asks banks to evidence traceability of risk data from source to report. GDPR requires organizations to map data flows for subject requests and privacy impact assessments. SOX requires audit trails for financial data pipelines. OJK in Indonesia and BNM in Malaysia require governance documentation that includes flow traceability. Automated lineage is the practical way to hold that position continuously rather than only at audit time.
It depends on the stack, and any page quoting a fixed number is guessing. The variables that decide it are how many source systems there are, how many of them have a native connector, whether the transformation logic is in a parseable tool or in hand written procedures, and how quickly domain owners can be named. Connected systems appear on the graph almost immediately, because capture is automated. The schedule is set by the systems that have no connector and by the reconciliation of duplicate business definitions the first honest graph exposes, and both of those are organizational rather than technical.
Yes, and it is increasingly required rather than optional. Lineage supplies the provenance chain that lets a data scientist certify which features entered a model, lets a supervisor understand the data basis for an automated decision, and lets a platform team detect when an upstream change has invalidated a model assumption. In retrieval augmented generation systems, lineage is how you keep the index limited to governed, verified assets.
Data observability monitors the health of pipelines as they run, detecting anomalies, schema drift, freshness failures and volume drops. Data lineage maps the structure of those pipelines, meaning which sources, transformations and destinations are connected. Observability answers whether the data is healthy now. Lineage answers where it came from and what it affects. Together they form the control layer for reliable data infrastructure.
First step to AI Readiness
Table of Contents
Sneak peek from the data world.
Bring lineage, quality, governance, and AI together in one platform—so your teams and AI agents can act with confidence.
_For%20light%20backgrounds.svg)
.svg){kind=small}
.webp)