Kindly fill up the following to try out our sandbox experience. We will get back to you at the earliest.
What column level data lineage is, how it is derived from SQL, where parsing fails, what it costs to run, and the impact analysis it makes possible.
Updated on
Column level lineage is a record of how each individual field in your data platform was produced: which source fields fed it, what expression combined them, and which reports, models and downstream fields read it afterwards. It is the same idea as data lineage and its types, resolved one level finer, at the field rather than at the table.
That extra level changes what the record can answer. Table level lineage supports the question "what depends on this table". Column level lineage supports "what produced this number", which is the question a finance controller, an engineer about to drop a column and a supervisor all happen to ask in slightly different words.
The rest of this page stays on the column level question specifically: the difference you can feel in a worked example, how the trace is derived and where the derivation gives up, what it costs to run, and the two jobs it genuinely pays for.
A finance team publishes a weekly revenue dashboard. On Tuesday morning the headline figure reads 4 percent lower than the same figure read on Monday. Nobody changed the dashboard. The question is what moved.
With table level lineage, the answer is that the dashboard reads from a revenue fact table, which reads from an orders staging table, a refunds staging table and a currency dimension. Three candidates, no ranking between them, and the next step is a person opening each transformation and reading the SQL by hand.
With column level lineage, the answer names the path. The dashboard field is the net revenue column of the revenue fact table. That column is the gross amount from orders minus the refund amount from refunds, multiplied by the conversion rate held in the currency dimension. Three fields feed it, and the conversion rate was reloaded at 02:00 on Tuesday while the other two were unchanged. The investigation is over before it started.
That is the whole argument for field level tracing, and it repeats in every job below.
| What the team is asking | Table level answer | Column level answer |
|---|---|---|
| What produced this revenue figure? | It depends on three upstream tables. | It is gross amount minus refund amount, converted at the stored rate, and each of those is a named field in a named table. |
| What changed on Tuesday? | One of the three upstream tables was written to. | Only the conversion rate field was reloaded. The other two inputs are byte identical to Monday. |
| What breaks if we drop this column? | Unknown. Every consumer of the table is a suspect. | The exact list of downstream fields, reports and models that read this field, and nothing else. |
| Which reports contain customer personal data? | Any report reading a table that holds a personal field. | Only the reports where a personal field actually flows through to an output column. |
| Can we prove this number to an auditor? | You can show the tables involved and then argue from the code. | You can show the derivation of the number as a path, with a date on every edge. |
| How is the answer produced? | A person opens each downstream query and reads it. | The graph is queried and returns the path. |
Accuracy work fails in a predictable way. A test fires on a table, someone confirms the table is wrong, and then a day disappears into working out which field carried the error and how far it travelled. Column level lineage removes that middle day, because the path from the failing field back to its sources is already recorded.
It also changes what monitoring is worth. A freshness or volume check on a table tells you the table arrived. A check on a column tells you the field that feeds your reported number is still within range, and the lineage graph tells you which reported numbers move when it is not. That pairing of per column checks with per column lineage is what turns data observability from an alert stream into an answer.
The honest limit is that lineage records derivation, not correctness. It will show you that a revenue figure came from a conversion rate field. It will not tell you the rate was wrong. Lineage narrows the search; the tests still have to exist.
Almost every column level graph in production is derived rather than declared. A tool collects the statements that ran, parses each one into a syntax tree, resolves the identifiers against the catalogue schema, and reads the column mapping out of the select list, the joins and the filters. Repeat that across every statement and the edges stitch into a graph.
Four sources feed that process, and most platforms combine them rather than picking one.
| Source of the trace | What it reads | Where it is strong | Where it stops |
|---|---|---|---|
| Query log parsing | The statements the warehouse actually executed | It sees what really ran, including ad hoc work and jobs nobody documented. | It only sees SQL that reached the warehouse, and it needs log retention long enough to cover infrequent jobs. |
| Transformation manifest | The compiled model graph a transformation framework produces | Precise on modelled pipelines, and it knows the intent behind each model. | It covers only what is inside that framework. Anything outside it is invisible. |
| Warehouse native metadata | Lineage the platform records about its own objects | Cheap, already there, and accurate inside one platform. | It ends at the platform edge, so the trace stops where the data leaves. |
| Declared mapping | A mapping a person registers by hand | The only option for a step no parser can read. | It goes stale silently. Every declared edge needs an owner and a review date. |
The practical consequence for a buyer is that "we have column level lineage" should be read as a coverage number over a defined scope rather than as a yes, and the scope is set by which of these four sources a tool actually reads. When you compare vendors, ask which sources they parse before you ask what the graph looks like. Our guide to the best data lineage tools works through that comparison in full.
No parser reaches complete coverage, and a vendor who says otherwise is describing a demo environment. The failures are well understood, they are the same ones in every platform, and each has a workable response. What separates a usable graph from a misleading one is whether the gaps are visible.
| Where the trace is lost | Why the parser cannot follow it | What to do instead |
|---|---|---|
| SQL assembled at runtime | The statement text is built by application code, so the column names do not exist until the moment of execution. | Parse the executed statement from the query log rather than the source code. What ran is always parseable, even when the code that produced it is not. |
| Stored procedures | A procedure holds branches, loops and temporary tables, so there is no single mapping from inputs to outputs to extract. | Parse the statements the procedure emits at runtime where the log captures them, and declare the input to output mapping by hand where it does not. |
| User defined functions | The function body may be written in another language and is opaque to a SQL parser. | Register the signature and treat every output as derived from every input. It is coarser than the truth but it never misses a dependency. |
| Wildcard selects | The parser cannot know which columns a star expands to without the schema as it stood when the query ran. | Resolve against the catalogue schema at parse time and reparse when the schema changes. Track these separately, because they are the edges most likely to be silently wrong. |
| Transformations outside the warehouse | Work done in Python, Spark jobs, notebooks or application code produces no SQL to read. | Instrument the job so it emits its own lineage events, or accept a declared edge and mark its confidence lower so nobody mistakes it for a parsed one. |
| Calculations in the reporting layer | The metric is defined inside the business intelligence tool, after the data has left the warehouse. | Pull the semantic model from the reporting tool and join it to the warehouse graph. Without this step the trace ends at the last table rather than at the number a person sees. |
The response to all six is the same discipline. Score every edge as parsed, declared or inferred, publish coverage as a number per table tier, and treat a low confidence edge as a known gap rather than as a fact. A graph that admits what it does not know is more useful than one that quietly guesses, because only the first one tells you where to look by hand.
Column level lineage is often sold on a long list of benefits. In practice two jobs pay for it, and the rest follow from those two.
An engineer is about to rename a field, change its type, or drop it. Table level lineage returns every consumer of the table, which on a wide table is most of the business, so the answer is unusable and the change either stalls or ships blind. Column level lineage returns the fields, reports and models that read that one field. The review becomes a short list you can send to named owners, and the pull request carries the blast radius with it.
A number is wrong and the pressure is immediate. Column level lineage turns the search from a breadth first scan of every upstream table into a walk backwards along the path that produced the field, with the last change on each edge attached. The Tuesday example above resolves in one traversal.
This is the question buyers ask and vendor pages avoid. A field level graph is more expensive than a table level one in every dimension, because the unit of work is the column and a warehouse has one or two orders of magnitude more columns than tables. The cost is manageable, but only if you know what drives it.
| Cost driver | Why it costs | How teams control it |
|---|---|---|
| Column count, not table count | The graph grows with fields and with every version of every field, so a 400 column table is not one node, it is 400. | Scope by table tier rather than by warehouse. Most teams find the tables that matter number in the dozens. |
| Volume of statements parsed | Every statement in scope has to be parsed, resolved against a schema, and reconciled with the existing graph. | Parse a rolling window of recent activity for the whole estate, and keep deep history only for the tables you would have to defend. |
| Schema change frequency | A rename or type change invalidates every edge that touched the field and forces a reparse of the statements behind it. | Reparse on a schedule and on schema change events rather than on every pipeline run. |
| History and versioning | Answering "what produced this number in March" needs the graph as it stood in March, which means storing versions rather than a current state. | Keep full version history for regulated and financial tables, current state only for the rest. |
| Traversal at query time | Deep impact analysis walks many edges, and an interactive graph over a large estate is a real query workload. | Precompute the traversals people actually run, usually one hop and full downstream from a field. |
The scoping rule that works is risk first. Start with the tables behind numbers you would have to defend to a regulator, an auditor or a board: reported financials, revenue, customer records carrying personal data. Add the tables with the most downstream dependents next, because those are where an unnoticed change causes the widest damage. Everything else can run at table level until it earns the upgrade. A team that starts by pointing a lineage tool at the entire warehouse usually ends up paying for a graph nobody reads.
The sequence below produces something people trust in weeks rather than a complete graph nobody has checked.
Supervisors rarely use the phrase column level lineage. They ask how a specific reported number was produced, who is accountable for the data behind it, and whether the controls held on the date it was reported. Each of those is a field level question, and a table level answer does not close it.
| Supervisor | Who it covers | What it asks that only a field level trace answers |
|---|---|---|
| OJK, Indonesia | Banks, insurers and financial technology firms | How a figure in a supervisory return was produced, and whether the data behind it was controlled. The figure is a field, not a table. |
| APRA, Australia | Banks, insurers and superannuation funds | Control over critical data elements with named accountability. A critical data element is a column, so the register and the lineage have to agree at that level. |
| MAS, Singapore | Financial institutions | Fairness, ethics, accountability and transparency for models that affect customers, which requires stating which input fields the model consumed. |
| NAIC, United States | Insurers, at state level | Documentation of underwriting and claims models. Rating variables are fields, and the question is where each one came from. |
This is where the cost argument usually settles. A team can debate whether field level tracing is worth the compute for a marketing dashboard. For a number that goes to a supervisor, the alternative to a trace is an engineer reconstructing derivations by hand under a deadline, which is slower, more expensive and far less convincing.
Decube derives column level lineage from the statements your warehouse actually ran and from your transformation models, then keeps per column checks on the same fields, so the graph and the monitoring agree rather than living in separate tools. Decube data lineage covers the trace itself, and the observability side adds the tests that tell you when a traced field has moved.
The part worth asking any vendor about, including us, is coverage. Ask which sources are parsed, what percentage of your scoped tables resolve to parsed rather than declared edges, and what the graph does with the transformations that never touch the warehouse. Request a demo if you want that answered against your own stack rather than against a sample one.
Column level lineage is a record of how each individual field was produced: which source fields fed it, what expression combined them, and which downstream fields, reports and models read it. It is data lineage resolved at the field rather than at the table, which is what lets it answer what produced this number rather than only what depends on this table.
Table level lineage records that one table feeds another. Column level lineage records which fields carry that connection and how they were combined. The practical difference shows up in impact analysis: table level lineage tells you a change might affect every consumer of a wide table, while column level lineage returns only the fields and reports that read the one field you are changing.
Decube provides column level lineage together with cataloguing and per column quality monitoring in one platform, which is what allows the lineage graph and the checks to run against the same fields. Several catalogue and observability vendors also offer it, at different coverage levels and with different pricing models. Our guide to the best data lineage tools compares them, including where each one derives its lineage from.
Yes, and the two are more useful together than apart. Cataloguing tells you a field holds customer data, lineage tells you every output field it reaches, and access control then applies to the whole propagated set rather than only to the original table. Without lineage, column level access control has to be maintained by hand on each table someone remembers.
They read the code rather than the finished tables: the compiled model graph from a transformation framework gives the column mapping, and tests defined alongside the models produce the quality metrics. The strength is precision on modelled pipelines. The limit is scope, because anything outside that framework, including work done in notebooks or application code, is invisible unless the warehouse query log is parsed as well.
Code level lineage traces data flow through application and transformation code rather than through executed SQL alone, which is how you cover steps written in Python, Spark or a stored procedure. It complements column level lineage rather than replacing it: the SQL parser produces the warehouse graph, and code level tracing fills the transformations the parser cannot read.
It needs it to do the two jobs metadata platforms are bought for. Impact analysis at table level returns every consumer of a table, which is too broad to act on, and root cause analysis without field level paths still ends with an engineer reading SQL by hand. Table level lineage is enough for a catalogue that documents; column level is what a platform needs to answer questions.
The argument above stays abstract until you open a column and look at what it actually feeds. This two minute walkthrough opens the column level view on a lineage graph in Decube, expands one column to its downstream mappings so you can read which specific fields connect rather than which tables, and hovers an edge to show whether that relationship came from a Snowflake query or a dbt job. It also switches on the incident and classification layers, so PII and open data quality issues appear on the same nodes you are tracing. Useful if you are about to decide which of your own fields earn a field level trace first.
First step to AI Readiness
Table of Contents
Sneak peek from the data world.
Bring lineage, quality, governance, and AI together in one platform—so your teams and AI agents can act with confidence.
_For%20light%20backgrounds.svg)
.svg){kind=small}
.webp)