Kindly fill up the following to try out our sandbox experience. We will get back to you at the earliest.
The four ways automated data lineage is captured, what each method resolves, where each one silently fails, and how to audit your real coverage.
Updated on
If you search for automated data lineage you will find a dozen pages explaining that it saves time compared with a spreadsheet. That is true and it is not useful. The question a data team actually has is narrower: how does a tool work out that this column came from that one, how much of my estate can it work out, and what does it quietly get wrong. This article answers those three.
For the definition itself and the vocabulary around it, our guide to data lineage concepts covers the ground properly. What follows assumes you already know what a lineage graph is and want to know how one gets built.
Automated data lineage is a lineage graph derived from metadata your platforms already produce, rather than a diagram someone drew. Both look identical on screen, so the difference that counts is where the edges come from. A manual graph has edges because a person asserted them. An automated graph has edges because a parser read a CREATE TABLE statement, or a warehouse logged a query, or a job announced what it wrote.
That difference decides everything else about the graph, including how fast it goes stale, how much of your estate it covers, and which parts of your pipeline it cannot see. A manual graph is wrong the day someone changes a model and forgets to update the diagram. An automated graph is wrong in a different way: it is wrong wherever the capture method has a blind spot, and it will not tell you where those are.
Every lineage product on the market is some combination of the four methods below. Vendors describe them with different words, and most will tell you they use "all of the above", but the underlying mechanics are these and the limits of each are fixed by physics rather than by product roadmap.
| Capture method | What it reads | What it resolves | What it cannot resolve |
|---|---|---|---|
| 1. SQL and code parsing | DDL and DML text, view definitions, dbt models, ETL scripts, stored procedure bodies | Table and column relationships before anything runs, including code that has never executed. Follows a chain of views through its middle. | SQL assembled at run time from a variable. Logic inside a user defined function. Any language the parser does not support. |
| 2. Query log and platform run history | The warehouse or platform record of statements that actually executed | What genuinely ran, including statements built at run time, plus who read what and when. | Anything outside the retention window. Work done outside the platform. Failed statements. Code that has never been executed. |
| 3. Metadata and API connectors | Manifests and APIs from dbt, orchestrators, ingestion tools and BI platforms | Declared dependencies, job to table mapping, dashboard to table mapping, and the semantic layer a warehouse never sees. | Dependencies the tool never declared. Logic computed inside a BI model. Anything with no API. |
| 4. Run time event emission | Events a job emits while it executes, usually to the OpenLineage specification | Spark, Python and streaming jobs that never emit SQL a parser can read. | Any job that has not been instrumented, which in most estates is most of them on day one. |
A parser reads your SQL as code. It builds a syntax tree, resolves each output column back through joins, common table expressions and subqueries, and produces edges without the statement ever running. This is the only method that gives you lineage for a model you wrote this morning and have not deployed, which is exactly the lineage you want before you change a schema.
Parsing is also the only method that reliably follows a chain of views through its middle. If a report reads View A, which reads View B, which reads View C, which reads the base table, a parser gives you all four nodes and the three edges between them.
It breaks on anything that is not there to read. A stored procedure that builds a statement by concatenating a table name onto a string has no statement in it, only the instructions for making one. A user defined function that transforms a value hides the mapping inside compiled logic. And a parser that supports six SQL dialects will silently produce nothing for the seventh.
Every serious warehouse records what it executed, and that record is the most honest lineage source you have, because it describes what happened rather than what was intended. In Snowflake this is the ACCESS_HISTORY view in the Account Usage schema. Its documentation, at docs.snowflake.com/en/sql-reference/account-usage/access_history, is worth reading in full before you rely on it.
The mechanics are specific enough to plan around. Access History requires Enterprise Edition or higher. It holds 365 days of history. Its latency may be up to 180 minutes, so the graph you look at this morning does not contain what ran an hour ago. It carries the objects a query named directly, the base objects needed to run it, the objects a write touched, and DDL changes, plus parent and root query identifiers so a statement that ran as a child job can be chained back to the job that launched it.
Column level lineage from that log is narrower than table level lineage, and Snowflake publishes the exact list. Column lineage is tracked for CREATE TABLE AS SELECT, CREATE TABLE CLONE, INSERT SELECT, MERGE, UPDATE in its two forms, and ALTER TABLE RENAME TO. If your pipeline writes some other way, the table edge appears and the column edges do not. Our explainer on column level lineage walks through what that granularity buys you when it is present.
The third method does not look at data movement at all. It asks the tools in your stack what they think they are doing. A dbt project already knows its own model graph. An orchestrator already knows which task feeds which. A BI platform already knows which dashboard reads which dataset. A connector pulls those declarations and stitches them together.
This is how lineage crosses a system boundary at all. A warehouse log cannot tell you that a table feeds a Tableau workbook, because from the warehouse side that is just another SELECT from another user. Only the BI tool knows what it did with the result.
The boundary is the tool own internal model. Tableau documents this plainly: for a connection built on custom SQL, "Lineage might not be complete", and Catalog "doesn't support showing column information for tables that it only knows about through custom SQL". The same shape of problem exists in every BI platform. A measure calculated inside the workbook is a transformation, and it is a transformation no warehouse and no parser will ever see.
The fourth method makes the job report on itself. A listener attached to the runtime emits an event when a job starts and finishes, naming the datasets it read and wrote. The open standard for this is OpenLineage, and its column lineage facet is precise about what an edge means: each output field carries the input fields that produced it, and each relationship is typed as DIRECT or INDIRECT, with a subtype such as IDENTITY, TRANSFORMATION or AGGREGATION for a direct one and JOIN, FILTER or SORT for an indirect one, a description, and a flag for whether the value was masked.
That vocabulary matters more than it looks. A column that only ever appears in a WHERE clause has an INDIRECT relationship to the output, and most capture methods drop it entirely. If you need to prove that a sensitive field influenced a published figure, the indirect edge is the one you need and the one you are least likely to have.
The cost of this method is deployment. Events only exist for jobs that have been instrumented, so coverage starts at zero and grows as your platform team works through the estate. It is the right answer for Spark and Python work that emits no readable SQL, and it is the wrong answer if you were hoping to switch it on for everything at once.
This is the table to look up your own situation in. Read down the left column until you find the thing your pipeline actually does, then read across to see which method will produce an edge for it. The query log column is anchored to the behavior Snowflake documents for Access History, because it is the platform that publishes its limits in most detail; other warehouses differ in the specifics but not in the pattern.
| Construct in your pipeline | Parsing | Query log | Connector | Run time events |
|---|---|---|---|---|
| CREATE TABLE AS SELECT | Yes | Yes | No | Yes |
| INSERT SELECT and MERGE into a target table | Yes | Yes | No | Yes |
| A view built on another view built on a base table | Yes | Partial | No | No |
| SQL assembled from a variable at run time | No | Yes | No | Yes |
| A stored procedure looping over a cursor | Partial | Yes | No | Partial |
| A user defined function that hides the column mapping | No | No | No | No |
| A column used only in a WHERE clause | Yes | No | No | Yes |
| A dbt model | Yes | Yes | Yes | Yes |
| A Spark or Python job that emits no SQL | No | No | Partial | Yes |
| A calculated field inside a BI workbook | No | No | Partial | No |
| A table referenced by storage path rather than by name | No | No | No | Partial |
| A CSV someone exported and emailed | No | No | No | No |
Two rows in that table deserve attention because they surprise people. A view on a view is resolved by a parser and only partly by a log: Snowflake records the query on the outer view and the base table, and not the views in between. And a column used only in a WHERE clause is the reverse case, invisible to the parser only when the parser is lazy, and invisible to Snowflake column lineage by design.
A missing edge does not announce itself. The graph renders, the nodes connect, and nothing on screen says "there were three more views here". These are the six failures that produce a confident, wrong graph, each one taken from the platform documentation rather than from experience alone.
Snowflake states this directly. For a query on View A where the structure is View A reading View B reading View C reading a base table, Access History records the query on View A and the base table, and not View B or View C. If your lineage comes only from the warehouse log, an entire layer of your semantic modeling is absent from the graph and nothing flags it. The fix is to run a parser over your view definitions as well as reading the log, which is why serious tools do both.
Snowflake documents this with an example. For the statement inserting into a column c1 by selecting c2 from table b where c3 is greater than 1, c2 is recorded as a source column for c1, and c3 is not recorded as a source column at all. That is correct as a description of data flow and misleading as a description of influence. If c3 is a customer identifier used to filter a published aggregate, the identifier shaped the output and your column lineage says it was never involved.
Native lineage is a rolling window, not an archive. The differences between platforms are large enough to change what you can answer.
| Platform | Documented native lineage retention | Other stated limits |
|---|---|---|
| Snowflake, ACCESS_HISTORY in Account Usage | 365 days | Enterprise Edition or higher. Latency up to 180 minutes. Failed queries are absent, though they appear in query history. |
| Databricks Unity Catalog | Rolling one year window | Nothing captured before 1 September 2024. No lineage for renamed catalogs, schemas, tables, views or columns, for RDDs, for global temp views, or for jobs submitted through runs submit or the spark submit task type. |
| Google Dataplex and BigQuery | 30 days | Column lineage is not collected for load jobs or routines, does not cover upstream lineage for external tables, and is limited to top level columns, so nested fields inside a STRUCT or JSON are excluded. |
Read those three rows together and the practical point is obvious. A quarterly reconciliation job leaves no trace at all in a 30 day window. An auditor asking what fed a figure published eighteen months ago is asking a question no native lineage on any of the three platforms can answer. If lineage is evidence, it has to be exported and kept, not queried live.
Databricks documents that lineage is not captured for renamed catalogs, schemas, tables, views or columns. Renaming is a normal part of refactoring, which means the ordinary act of tidying a model can sever the recorded history of the thing you tidied. The asset still exists, the data still flows, and the graph now shows an origin that starts at the rename.
Two documented cases sit side by side in the Databricks limitations. Column lineage is not captured when sources or targets are referenced as a storage path rather than a table name, and user defined functions obscure the mapping from source columns to target columns. Both are common in teams that came to the platform from a data lake background, and both produce a table edge with no column detail underneath it.
Databricks states that lineage events persist even if the transaction is rolled back. The graph can therefore show a relationship that never actually landed in a table. This is the failure that matters most for an audit, because it is the one where the lineage record is not incomplete but wrong, and a reviewer reading the graph has no way to tell the difference.
Every vendor will quote you a coverage figure. It is always coverage of the assets that vendor can see, which is a different number from coverage of the assets you have. Here is how to produce your own, in an afternoon.
| Coverage number | How to calculate it | Which method a low score points at | Decube working target |
|---|---|---|---|
| Asset coverage | Assets with at least one resolved upstream or downstream edge, divided by assets in scope | Connectors, if whole systems are absent. Parsing, if only views and models are missing. | 95 percent inside a single warehouse |
| Column coverage | Columns with a resolved source column, divided by columns in scope | Parsing, and the write operations your pipeline uses that the platform does not track at column level | Measured and reported separately, never assumed to match asset coverage |
| Cross system coverage | Edges that cross a platform boundary, divided by the boundary crossings you know exist | Connectors and run time events, since a warehouse log cannot see past its own edge | 80 percent across an estate spanning ingestion, warehouse and BI |
The two target figures above are Decube working targets, offered as an operating rule for a team that has nothing to aim at. They are not a survey result and no research firm published them.
The short walkthrough above shows the export step in practice: the lineage graph leaves the platform as a flat CSV in which every node, edge and classification becomes a row, the file is filtered on the classification field to isolate every column marked as PII, and the from asset and to asset columns are read to trace source tables through the named ETL job to the final aggregate. That file is what you hand a regulator. The canvas on your screen is not.
Automation produces the graph. It does not produce the judgment about the graph, and the difference is where most lineage programs stall.
The practice side of this, the standards, the review cadence and what a complete lineage record has to contain, is covered properly in our data lineage best practices guide, which is the companion to this page rather than a repeat of it.
Answered on capture scope, which is the only honest way to answer it. Snowflake native lineage is query log capture, and it is genuinely good at what it does. It knows what ran inside Snowflake for the last 365 days, it produces column level detail for the operations listed earlier, and it costs nothing extra on Enterprise Edition or higher.
What it does not do is any of the other three capture methods. It does not parse your dbt project as code, so it cannot show you a change before it runs. It does not read your ingestion tool, so lineage begins at the moment data landed rather than at the source system. It does not follow a column into a workbook, so the report layer is outside the graph. And it records the outer view and the base table without the views in between.
The decision rule is therefore simple. If your estate is one Snowflake account and your reporting lives in Snowsight, native lineage is enough and a second tool is an expense without a purpose. If a single lineage question has to cross an ingestion tool, a warehouse and a BI platform, which is what a regulator question usually looks like, you need something that runs all four capture methods and joins the results. That is the entire case for a dedicated data lineage tool, and it is worth nothing if your estate does not have those boundaries.
These are the outcomes teams buy automated lineage for. They are worth stating plainly, with the caveat that every one of them depends on the coverage you measured above rather than on the coverage you were sold.
When a metric is wrong, lineage turns "which of the forty upstream tables broke this" into a list of three. Showing where a value came from and what changed it is what makes a quality problem findable rather than arguable.
A regulator asking where a reported figure came from is asking for a lineage record. An automated one is generated as a side effect of running the business, which is why it holds up better than documentation maintained by hand. It only holds up if it was exported before the retention window closed.
A shared view of which assets exist and who is accountable for them is what lets a data steward make a decision without convening a meeting. Lineage is the map that makes the ownership register meaningful.
Impact analysis before a schema change is the highest value use of lineage and the one most teams reach for first. Knowing every downstream table, job and dashboard that depends on a column turns a risky change into a scheduled one.
Engineers, analysts and business users arguing about a number are usually arguing because they hold different mental models of where it came from. A single graph replaces those with one model that anyone can open.
Manual lineage gets worse as the estate grows, because the documentation burden grows faster than the team. Automated capture is the only version that survives a doubling of your pipeline count, which is the practical argument for it.
A feature list tells you very little. The question worth asking is which of the four capture methods a tool actually runs, and for which of your systems. Ask a vendor that question system by system and the shortlist gets short quickly.
| Tool | Primary capture methods | Best fit |
|---|---|---|
| 1. Decube | Parsing, query log, connectors | Teams that need lineage to cross ingestion, warehouse and BI, and to leave the platform as audit evidence. |
| 2. Atlan | Query log, connectors | Organizations that want a collaborative catalog with lineage attached and a wide connector list. |
| 3. Secoda | Query log, connectors | Smaller teams that want discovery, documentation and lineage in one place without a long rollout. |
| 4. DataGalaxy | Connectors, query log | Teams whose main problem is the business glossary and the mapping between it and physical assets. |
| 5. IBM Manta Data Lineage | Parsing, connectors | Large estates with legacy ETL and stored procedure logic that only a code scanner will read. |
Decube is a cloud native platform that automates discovery and lineage mapping across the systems it connects to, and it is built for estates where lineage has to cross a boundary rather than stay inside a warehouse. It reads warehouse query history, parses transformation code, and pulls declarations from ingestion tools and BI platforms, so the graph carries edges from more than one method rather than resting on one.
What it does with the graph once built is the part that matters for a governance team. Lineage exports to a flat CSV carrying every node, edge and classification, so evidence survives outside the platform and past a warehouse retention window. Governance policies are configurable against the regimes a team actually reports under, and Decube works with organizations regulated by OJK in Indonesia, APRA in Australia, MAS in Singapore and NAIC in United States insurance, alongside GDPR and HIPAA obligations. Pricing is published rather than quoted on request: Starter is 175 USD per user per month from 21,000 USD a year with a ten user minimum, and Growth is 225 USD per user per month from 54,000 USD a year with a twenty user minimum, listed on the Decube pricing page.
Best for organizations that need automated lineage and governance in one place, particularly in finance, healthcare and commerce, where the lineage record has to be produced for someone outside the data team.
Atlan is a data governance platform with strong lineage and a collaborative interface that works for technical and non technical users alike. It gives a wide view of data flows across a connected stack and applies AI to surface context around assets, and its connector list is one of the longest available.
Best for organizations that want lineage as part of a broader catalog and collaboration workflow, and that value breadth of integration over depth of code parsing. Its own published guidance on automated lineage, at atlan.com/automated-data-lineage/, is a good statement of the benefits case, though it does not describe the capture mechanics.
Secoda focuses on discovery and cataloging with automated lineage attached. It connects a range of sources, maps movement between them, and presents the result in an interface aimed at analysts rather than platform engineers.
Best for teams that want a short rollout and a single place to search, document and trace, and that are working mostly inside a modern warehouse rather than across legacy systems.
DataGalaxy is a data intelligence platform whose lineage tracking is tightly coupled to its glossary and catalog. It maps the path from origin to destination and reports on it in a form aimed at data stewards and business owners, with reporting features built around regulatory obligations such as GDPR.
Best for organizations whose main gap is the connection between business definitions and physical assets, and who want lineage presented in business terms.
MANTA is now sold by IBM as IBM Manta Data Lineage, described in the IBM product documentation as a lineage service that increases pipeline transparency. Its strength has always been the breadth of what it can read as code, including older ETL platforms and stored procedure logic that a warehouse log will never explain.
Best for large enterprises with a long tail of legacy transformation logic, where the lineage problem is genuinely a code scanning problem rather than a connector problem.
Add these five questions, which come straight from the failures earlier in this article and which most vendors are not asked: which capture methods do you run for each of my named systems, do you parse code as well as read logs, what happens to your column lineage on a MERGE and on a user defined function, can I export the whole graph with classifications, and how far back does lineage go and does it survive a rename.
A rollout that tries to cover everything at once produces a graph nobody trusts. These four practices still hold.
Start with one domain where a specific question is already being asked, prove the coverage number on it with the audit method above, and expand from there. If you want to see what that looks like against your own stack, request a demo and bring the twenty assets you already understand.
Automated data lineage is a lineage graph built from metadata your platforms already produce, rather than one a person draws and maintains. Its edges come from a parser reading transformation code, a warehouse logging the queries it executed, a connector reading a tool declared dependencies, or a job emitting an event about what it read and wrote.
Four ways. SQL and code parsing reads transformation code and resolves relationships before anything runs. Query log capture reads the platform record of statements that actually executed. Metadata and API connectors pull declared dependencies from dbt, orchestrators, ingestion tools and BI platforms. Run time event emission has instrumented jobs report what they read and wrote, usually to the OpenLineage specification. Most estates need at least three of the four.
Data flow is the movement itself, the actual passage of records from one system to another as a pipeline runs. Data lineage is the recorded relationship between the assets that movement creates, held as a graph you can query after the fact. A pipeline has data flow whether or not anyone is capturing it; it has lineage only when something records where each output came from. Flow is an event, lineage is the evidence about it.
Snowflake native lineage is query log capture and it covers what ran inside Snowflake for the last 365 days, with column level detail for a defined list of operations. It does not parse your dbt project as code, does not reach back into an ingestion tool, does not follow a column into a BI workbook, and records the outer view and the base table without the views in between. If your estate is one Snowflake account and your reporting lives in Snowsight, it is enough. If a lineage question has to cross an ingestion tool, a warehouse and a BI platform, you need a tool that runs all four capture methods and joins the results.
For a regulated financial services firm the deciding criteria are cross system coverage, column level detail, and whether the lineage record can leave the platform as evidence. Decube is built for that case: it combines parsing, query log capture and connectors so lineage crosses ingestion, warehouse and BI, exports the full graph with classifications to CSV so evidence survives a retention window, and is used by organizations reporting to OJK in Indonesia, APRA in Australia, MAS in Singapore and NAIC in United States insurance. IBM Manta Data Lineage is the stronger choice where the estate is dominated by legacy ETL and stored procedure code that only a scanner will read, and Atlan is a reasonable choice where the priority is catalog breadth rather than lineage depth.
Tracing a wrong number back to its source instead of searching for it, producing a lineage record for a regulator as a side effect of running the business, giving governance a shared map with clear ownership, running impact analysis before a schema change rather than after an incident, giving engineers and analysts one model of where a figure came from, and scaling without adding documentation work as the pipeline count grows.
Impact analysis before a schema or model change, root cause analysis when a metric moves unexpectedly, proving to an auditor or regulator how a reported figure was produced, tracing personal data through a pipeline for a privacy request, deciding whether a table is safe to deprecate, and onboarding a new analyst who needs to know what feeds what.
Because every decision about data depends on knowing where it came from. Without lineage, a wrong number is a search, a schema change is a risk, a privacy request is a manual investigation, and an audit is a reconstruction. Lineage turns each of those from an investigation into a lookup, provided the coverage is real and has been measured.
More than one capture method, because no single method covers an estate. Column level lineage as well as table level, reported as a separate coverage number. Coverage that crosses system boundaries from ingestion through the warehouse to BI. An export that takes the whole graph with its classifications out of the platform, so evidence outlives the retention window. And a way to declare the assets no connector can reach, such as a spreadsheet or a vendor file drop.
Data lineage tracking is the ongoing capture of relationships between data assets as the estate changes, as opposed to a one time mapping exercise. Tracking implies the graph updates itself when a model changes, which is only true within the limits of the capture methods in use and the retention window of the platform underneath.
No, and a tool reporting 100 percent is reporting coverage of the assets it is connected to rather than the assets you own. Documented gaps exist on every platform: renamed objects, tables referenced by storage path, user defined functions, columns used only in a filter, jobs that were never instrumented, and anything living in a spreadsheet or a file drop. Measure your own number by sampling twenty assets you already understand and tracing each one by hand.
First step to AI Readiness
Table of Contents
Sneak peek from the data world.
Bring lineage, quality, governance, and AI together in one platform—so your teams and AI agents can act with confidence.
_For%20light%20backgrounds.svg)
.svg){kind=small}
.webp)