Kindly fill up the following to try out our sandbox experience. We will get back to you at the earliest.
Data freshness is how old the newest record in a table is right now. What it means, four ways to measure it, how to set a threshold, and when stale is fine.
Updated on
"Data is a precious thing and will last longer than the systems themselves." - Tim Berners-Lee
Data starts ageing the moment it is written, and the volume and complexity of a modern stack make it harder to tell how old any given table has become. This page answers the four questions a data team actually has about that: what data freshness means, how it is measured, what threshold to set, and when a stale table is fine to leave alone.
Data freshness is the age of the newest record in a dataset, measured from now. If the most recent row in an orders table carries a timestamp of 09:12 and the clock says 09:41, the freshness of that table is 29 minutes.
Written as a calculation it is one line: freshness equals the current time minus the maximum timestamp in the table. The timestamp can be the moment the event happened, the moment the row was loaded, or the moment the table was last written to, and which one you pick changes the answer. That choice gets its own section below, because it is where most freshness monitoring quietly goes wrong.
Freshness describes the dataset, not the decision. It tells you how old the data is. Whether that age is acceptable is a separate question, and it depends entirely on what the data is used for. A customer table refreshed once a day is fresh enough for a monthly board pack and far too old for a fraud check on a card transaction.
A decision made on stale data is a decision made about a world that has moved on. The retail example holds well: a store that reorders on last year's sales figures buys too much of what sold then and too little of what sells now, and pays for the difference twice, once in dead stock and once in the sales it could not fill.
Three kinds of decision break on stale data in ways a team can put a number against. Pricing and inventory decisions taken on yesterday's positions. Credit and fraud decisions taken on a customer record that predates the event being checked. And any decision an automated system takes without a human reading the timestamp first. The third is the expensive one. A person glancing at a dashboard often notices that a figure looks older than it should. A model scoring a transaction never does.
These four terms get used as though they were interchangeable, and the confusion has a practical cost: a team that cannot say which one it is measuring cannot agree on whether there is a problem. Each measures something different and each is expressed differently.
| Term | What it measures | What it is a property of | How it is expressed |
|---|---|---|---|
| Data freshness | How old the newest record in a dataset is, right now | The dataset | A duration that grows every second the table is not updated, such as 29 minutes or 4 hours |
| Data latency | How long one record takes to travel from the event happening to being queryable | The pipeline | A duration per record, usually reported as a percentile such as p99 |
| Data timeliness | Whether the data arrived before the decision that needed it | The use of the data, not the data itself | A yes or no verdict against a deadline |
| Stale data | Freshness that has crossed an agreed threshold | The dataset, judged against a policy | A verdict plus the amount it is over by |
| Data recency | The timestamp of the last update itself | The dataset | A fixed point in time rather than a growing duration |
Latency asks how long a single record takes to get from the event happening to the row being queryable. Freshness asks how old the newest record you can query is right now. They move together most of the time, which is why they get conflated, but a pipeline can be low latency and stale at the same moment. If your streaming pipeline moves each record in 400 milliseconds and the source system stopped emitting events an hour ago, latency is 400 milliseconds and freshness is 60 minutes. Latency measures the pipeline. Freshness measures the dataset sitting at the end of it.
Timeliness is a judgment rather than a measurement. It asks whether the data was there in time for the decision that needed it. The same table can be timely for one consumer and late for another on the same morning: a finance close that starts at 10:00 is served perfectly by a table landing at 09:30, while a trading desk that opened at 08:00 was not. This is why a freshness figure on its own never tells you whether you have a problem. You need the consumer threshold beside it, and the threshold comes from the consumer rather than from the platform team.
Stale is what a dataset becomes once its freshness crosses a line somebody drew. Without that line the word carries no information, which is why a team arguing about whether a table is stale is usually arguing about a threshold nobody ever wrote down. Set the threshold first and the argument becomes a number that either is or is not exceeded.
Recency is the timestamp itself. Freshness is the distance from that timestamp to now. The two are used interchangeably in most vendor documentation and the difference rarely matters, with one exception that does: a recency value stays fixed while a freshness value grows every second the table is not updated. Alerting has to work on the second one.
Four measurement methods are in common use. Most teams run the first, should also run the fourth, and get the largest reduction in noise from the third.
The standard check. Take the maximum timestamp in the table, subtract it from the current time, and compare the result against a threshold. In SQL it is a single expression: TIMESTAMPDIFF(HOUR, MAX(last_updated), CURRENT_TIMESTAMP()). It is cheap, it runs everywhere, and it is the method behind most freshness alerts in production today. Its weakness is that it trusts whatever column you point it at, which is the trap covered further down.
This measures the gap between when a record existed in the source system and when it became queryable in the warehouse. It needs each record to carry more than one timestamp: the time the event occurred, the time it was ingested, the time it was transformed, and the time it became available to query. With those four stamps you can say which stage added the delay rather than knowing only that the table is late. It answers the question a plain freshness alert cannot, which is not whether the data is late but where it got stuck.
Instead of a fixed number, the monitor learns how a table normally behaves and raises an incident when the pattern breaks. A table that has landed between 02:10 and 02:25 every night for three months does not need a human to pick a threshold, and a load that has not arrived by 02:40 is already anomalous. This is how the freshness monitor in Decube works: it learns each table's own update pattern rather than firing against a number typed in once and never revisited.
The reason this matters is noise. A static threshold is either tight enough to fire on every ordinary variation or loose enough that a genuinely broken table waits half a day for its alert. Monitors that fire constantly get muted, and a muted monitor protects nobody.
The remaining two methods compare a table against something outside itself. A heartbeat is a row the pipeline writes on every run whether or not there was data to load, so an empty load and a failed load stop looking identical. A cross dataset check compares tables that should move together: if orders landed and payments did not, the payments table is late even when its own timestamp still sits inside its normal window.
This is the failure that survives most freshness programs. If your check reads a loaded_at column that the pipeline sets on every run, the table reports itself fresh for as long as the job keeps running, even after the source system stopped sending anything. The job succeeded, the column updated, the monitor stayed green, and the table has been serving the same rows for two days.
The fix is to measure against a timestamp the source controls rather than one your own pipeline writes. Where that is impossible, pair the freshness check with a row count check, because a job that loads zero new rows and still stamps loaded_at is exactly the case a volume monitor catches and a freshness monitor misses.
A freshness threshold is the age at which data stops being acceptable to its consumers. It is the number that turns freshness from a measurement into an alert, and skipping it is how you end up with monitoring that reports an age with nothing to compare it against.
Derive it from three inputs added together: the interval the table is expected to update on, one full retry cycle of the job that loads it, and the time the owning team needs to notice and act before a consumer is affected.
| Input to the threshold | Where the number comes from | Worked example, hourly orders table |
|---|---|---|
| Expected update interval | The schedule the table is actually loaded on, read from the orchestrator rather than from what somebody remembers | 60 minutes |
| One full retry cycle | The orchestrator retry policy for the job that loads the table | 15 minutes |
| Time to notice and act | How long the owning team needs to see the alert and fix the load before a consumer is affected | 45 minutes |
| Warning threshold | The three rows above, added together | 2 hours |
| Failure threshold | The point at which a consumer is definitely affected and the incident stops being the pipeline team's alone | 4 hours, ahead of the 09:00 analyst start |
The example above produces two hours, not one. Setting the warning at one hour on an hourly table means every ordinary retry pages somebody, and a monitor that pages somebody for a non event is a monitor with a short life expectancy.
Most tables fall into one of five tiers. The tier is set by the consumer, not by what the pipeline happens to be capable of.
| Tier | Freshness target | What it is normally used for | What it demands of the pipeline |
|---|---|---|---|
| Real time | Under a few seconds | Fraud scoring, trading, live pricing and personalisation | Streaming ingestion, continuous processing, and monitoring that itself runs in seconds |
| Near real time | Under a few minutes | Operational dashboards, stock and inventory levels, on call alerting | Micro batch loads or change data capture, with a retry policy measured in seconds |
| Hourly | Under 60 minutes | Campaign reporting, support queues, sales pipeline views | Scheduled batch loads with room for at least one retry inside the hour |
| Daily | Under 24 hours | Executive reporting, finance dashboards, most model training sets | One nightly window with room for a single rerun before anyone opens a dashboard |
| Weekly or monthly | Days | Regulatory submissions, board packs, slowly changing reference data | A calendar, a named owner, and a check that the load ran at all |
Moving a table up a tier costs engineering time and an on call rota rather than storage. The honest question to ask before promoting a table to near real time is who gets woken at 03:00 when it breaks, and whether the decision that table feeds is one anybody actually makes at 03:00.
A single threshold gives you a binary, and a binary makes every freshness problem an emergency. Two thresholds give you a window in which somebody can fix the load before anyone downstream notices.
The source freshness configuration in dbt is the clearest published example of the pattern. Its documentation for the freshness property defines warn_after and error_after, each taking a count and a period of minutes, hours or days, and the example it gives warns at 12 hours and fails at 24 for a source expected to land daily. One detail in that documentation is worth copying into any freshness check, whatever tool you run it in: if neither warn_after nor error_after is set, dbt does not calculate freshness for that source at all. A source with no threshold set is not being watched leniently; nothing is watching it.
The same documentation carries a second detail that connects directly to the trap above. You can point the check at a specific column with loaded_at_field, or, on Snowflake, Redshift, BigQuery and Databricks, let it read the warehouse metadata instead. Reading warehouse metadata is cheaper and it is also the version most exposed to a false green, because warehouse metadata records when the table was written rather than when the data was born.
A freshness monitor is a threshold plus a schedule plus somebody who receives the alert. In production the difficulty is rarely the detection. Deciding which of two thousand tables deserve a monitor, who owns each one, and what breaks downstream when one goes late is the part that takes a quarter rather than an afternoon.
The three minute walkthrough below goes through each monitor type in Decube and, more usefully, how to choose between them: schema drift and job failure enabled automatically as soon as a source is connected, freshness and volume monitors that learn each table's update pattern instead of firing on a fixed number, field health checks for nulls, uniqueness and regex compliance at column level, custom SQL for cross table business rules, and group by monitoring that pinpoints the dimension where quality broke.
Rolling monitors out across a warehouse is a different job from understanding the metric, and our guide to freshness monitoring best practices for data engineers covers the rollout in detail: which tables to start with, how to assign owners, and how to stop coverage growth turning into alert volume growth.
If you are wiring monitors into a pipeline rather than clicking through a console, the Decube public API creates, updates and deletes data quality monitors programmatically, freshness monitors included, so a table's threshold can live in the same repository as the job that has to meet it.
Not every table needs to be fresh, and treating them as though they do is the fastest route to a monitoring system nobody reads. Five cases where stale is the correct state:
The general rule: monitor freshness where a stale value would change a decision. Everywhere else, monitor that the load ran at all and leave freshness out of it.
Four broad factors set how fresh a dataset can be: where the data comes from, how often it is collected, how it is stored, and how long it takes to process. Those are the categories. In production the causes are more specific than that, and five of them account for most incidents.
Storage belongs on the list too. A table that has been corrupted, partially restored, or rebuilt from a backup can carry timestamps that no longer describe when the data was current, so any restore should be followed by a freshness check rather than assumed to have fixed one.
Four methods, in the order they usually pay off.
For teams answering to OJK in Indonesia, APRA in Australia, MAS in Singapore or the NAIC in the United States, freshness carries an evidence requirement on top of the operational one. Having a current number is only half of it. You have to be able to show afterwards how current it was at the moment a decision was taken, which means the freshness measurement itself has to be retained, not only the alert it did or did not raise.
Privacy concerns and regulatory compliance apply to the freshness project itself, not only to the data inside it. Moving a personal data table from daily to streaming changes how long that data sits in intermediate systems and who can see it there.
Freshness is one dimension of data quality alongside completeness, accuracy, uniqueness, validity and consistency, and it is one of the signals a data observability platform watches alongside volume, schema and lineage. It gets more attention than the other dimensions for a practical reason: it fails visibly. A number that has not moved since yesterday is obvious to a business user in a way that a subtly wrong number is not, so freshness is usually the first data quality problem a team gets asked about and the first one it instruments.
If you are working out how those two disciplines fit together, data quality and data observability covers the split between them, and data quality management covers the wider program that freshness monitoring sits inside.
Freshness gets harder the more downstream tables depend on a source, which is the case the data observability module in Decube was built for. Freshness monitors learn each table's own update pattern rather than running against a number typed in once, so an alert reflects how that table actually behaves rather than how somebody expected it to behave in a planning meeting. When a table does go late, the incident view shows which downstream tables, dashboards and models it touches before anybody has to work that out by hand.
Pricing is published rather than quoted. Starter is 175 USD per user per month, from 21,000 USD a year with a minimum of 10 users, and Growth is 225 USD per user per month, from 54,000 USD a year with a minimum of 20 users.
The fastest way to know what your own freshness picture looks like is to see it on your own tables. Request a walkthrough and bring the ten tables whose staleness would change a decision.
The thing that has genuinely changed about freshness since this article was first written is who reads the data. A person opening a dashboard often notices that a figure looks older than it should and goes to ask. An agent querying a table through an API or an MCP server does not. It answers on whatever it finds, and it answers with the same confidence either way.
That moves freshness from a reporting concern to an interface concern. If a table will be read by something that cannot judge whether a number looks stale, the freshness state has to travel with the data, published as metadata the consumer can read rather than only as an alert sent to the team that owns the pipeline.
The practical version, and a task worth doing this week: pick the ten tables whose staleness would change a decision, write down the threshold each one's consumers actually need, and check what your current monitoring would do if each of those tables stopped updating tonight. Most teams find at least one that would stay green.
Data freshness is the age of the newest record in a dataset, measured from now. If the most recent row in an orders table carries a timestamp of 09:12 and the clock says 09:41, the freshness of that table is 29 minutes. Written as a calculation it is the current time minus the maximum timestamp in the table. Freshness describes the dataset. Whether the age it reports is acceptable is a separate question that depends entirely on what the data is used for.
Four methods are in common use. The first is the timestamp differential: the current time minus the newest timestamp in the table, which in SQL is a single expression such as TIMESTAMPDIFF(HOUR, MAX(last_updated), CURRENT_TIMESTAMP()). The second is source to destination lag, which needs each record to carry an event time, an ingestion time, a processing time and an availability time so you can see which stage added the delay. The third is a learned update pattern, where the monitor learns how the table normally behaves and raises an incident when the pattern breaks. The fourth is a heartbeat row or a cross dataset check, which compares the table against something outside itself so an empty load and a failed load stop looking identical.
Data freshness monitoring is the practice of checking the age of a table continuously against a threshold and raising an incident when it goes over. In production it is mostly a coverage problem rather than a detection problem: detecting that one table is late is easy, while deciding which tables deserve a monitor, who owns each one and what breaks downstream when one goes late is the hard part. Monitors that use a learned baseline rather than a fixed number stay useful longer, because a monitor that fires on ordinary variation gets muted and a muted monitor protects nothing.
Add three numbers together: the interval the table is expected to update on, one full retry cycle of the job that loads it, and the time the owning team needs to notice and fix a broken load before a consumer is affected. An orders table that loads hourly, retries once after 15 minutes and needs 45 minutes of response time gets a warning threshold of two hours, not one. Set a warning and a failure threshold separately, so there is a window in which somebody can fix the load before it becomes an incident. The dbt documentation uses a warning at 12 hours and a failure at 24 hours as its example for a source expected to land daily.
Latency measures how long a single record takes to travel from the event happening to the record being queryable, so it is a property of the pipeline. Freshness measures how old the newest record you can query is right now, so it is a property of the dataset. A pipeline can have excellent latency and terrible freshness at the same time: if each record moves in 400 milliseconds but the source system stopped emitting events an hour ago, latency is 400 milliseconds and freshness is 60 minutes.
Freshness is a measurement and timeliness is a judgment. Freshness gives you a number, the age of the data. Timeliness asks whether that number was small enough for the decision that needed the data, which means the same table can be timely for one consumer and late for another on the same morning. A table that lands at 09:30 serves a finance close starting at 10:00 perfectly and fails a trading desk that opened at 08:00. This is why freshness on its own never tells you whether you have a problem: you need the consumer threshold as well.
Stale data is data whose freshness has crossed a threshold somebody agreed to. Stale is a verdict rather than a metric, which is why the word carries no information until the threshold exists. A team arguing about whether a table is stale is usually arguing about a threshold nobody ever wrote down, and setting the threshold turns the argument into a number.
In five situations. A closed accounting period, where the numbers are meant to stop moving. A frozen training or evaluation set, where reproducibility depends on the data not changing. Slowly changing reference data such as country codes, currency lists and product hierarchies, which legitimately update a few times a year. A table whose only consumer runs monthly, because thresholds come from consumers. And an immutable event log or archive, where what needs watching is whether new rows are still arriving, which is a volume check rather than a freshness one. The general rule is to monitor freshness where a stale value would change a decision, and everywhere else monitor only that the load ran.
Either by querying a timestamp column in the table or by reading the warehouse metadata that records when the table was last written. Snowflake, Redshift, BigQuery and Databricks all expose that metadata, and reading it is cheaper than scanning the table. It also carries the largest trap in warehouse freshness monitoring: metadata records when the table was written, not when the data was born, so a load job that runs on schedule and writes zero new rows keeps the table reporting itself fresh. Where you cannot measure against a timestamp the source controls, pair the freshness check with a row count check, because a volume monitor catches exactly the case a freshness monitor misses.
Because lead data decays faster than almost any other business data and it decays invisibly. A lead record that is a day old has usually already been contacted by somebody else, and contact details go out of date at a rate that makes a weekly refreshed CRM confidently wrong rather than merely incomplete. Freshness in lead data means two separate things that both have to be tracked: how recently the lead took an action, and how recently the record itself was verified.
Domain data freshness is freshness measured and owned per data domain rather than centrally. In a domain oriented architecture the team that produces a dataset publishes the freshness guarantee for it, in the same way it publishes the schema, and consumers in other domains hold that team to the published number rather than to a platform wide default. It works because the domain team is the only group that knows what its source system can actually deliver, and it fails when nobody publishes the number, because a threshold with no owner is a threshold nobody defends.
First step to AI Readiness
Table of Contents
Sneak peek from the data world.
Bring lineage, quality, governance, and AI together in one platform—so your teams and AI agents can act with confidence.
_For%20light%20backgrounds.svg)
.svg){kind=small}
.webp)