It was nine on a Monday morning when the sales report came up as zero. It had not been a bad weekend: it was a file that never arrived. The source failed overnight, the pipeline ran anyway over empty data, and the dashboard showed, with complete confidence, a wrong number. The person who noticed was the sales director — not the data team.
This scene repeats itself in many organisations and explains why data observability has stopped being a luxury. As data passes through more stages — ingestion, transformation, models, reports — the points where something can break silently multiply. And the most expensive problem is not the pipeline that fails loudly; it is the one that keeps running and producing plausible but wrong numbers.
Data observability is the practice of knowing, at any moment, whether the data you rely on is healthy — and of being alerted when it is not, before the wrong decision has already been made. This article explains the concept, why traditional tests are not enough, the five pillars that make it up and how to start without having to buy a platform straight away.
What data observability is
The term comes from software engineering. In application systems, observability is the ability to understand a system's internal state from the signals it emits: metrics, logs and traces. Data observability takes that idea and, instead of watching CPU usage or request latency, watches the health of the data itself and of the pipelines that move it.
The principle is the same: instrument the system so that it tells you what is happening, and detect deviations early. Applied to data, it means continuously monitoring signals such as the time a table was last updated, how many rows it received, whether the schema changed and whether the distribution of values still looks like it usually does. When any of those signals departs from the pattern, someone is alerted — ideally before the report.
Why tests are not enough
Many teams already write data tests: a column cannot have nulls, a key must be unique, a total must add up. They are valuable and should exist. But tests check what we already know can go wrong — the so-called known unknowns. We write an assertion because we anticipated the problem.
Reality mostly produces unknown unknowns: the failure nobody foresaw. A source that changes a date format, a supplier that starts sending values in uppercase, a join that begins duplicating rows after an upstream change. There is no test written for that, because nobody imagined the scenario. Observability covers that territory: instead of only asking whether the data passes the rules we wrote, it asks whether it is different from what it usually is. Tests and observability do not compete — they complete each other.
The five pillars of data observability
A widely used way of organising the topic describes five pillars. Together, they cover most of the symptoms of unhealthy data:
- Freshness — is the data up to date? When did the table last receive new rows?
- Volume — is the amount within expectations? A day with half the usual rows, or double, is suspicious.
- Schema — has the structure changed? A column that disappears, changes type or is renamed breaks everything downstream.
- Distribution — are the values still plausible? A spike in nulls, a field that suddenly holds only zeros or an average that jumps reveal problems that a row count does not catch.
- Lineage — how do tables and reports connect to each other? It is the map that lets you go from symptom to cause and measure the impact.
No pillar is enough on its own. A table can be fresh and have the right volume, yet be full of nulls because of a change in the source schema. It is the combined reading that builds trust.
Freshness and volume: the earliest warnings
Freshness and volume are often the most useful alarms because they catch the most frequent failures — data that did not arrive or arrived short. Monitoring freshness means comparing the time of the last update with what you would expect: if a table is usually loaded by 6 a.m. and at 8 a.m. it still holds yesterday's data, something failed.
Volume adds a second reading. A pipeline can run on time and still bring only a fraction of the rows — for example, because the source exported a single region. Defining bands of normality for the daily row count, with room for natural seasonality, turns a dangerous silence into an actionable alert. These two signals alone prevent a considerable share of the classic incidents.
Schema and distribution: when the shape of the data changes
Schema changes are among the most treacherous causes of breakage. Someone, three teams upstream, renames a column or changes its type, and the pipeline either breaks or — worse — fills the field with nulls without complaining. Watching the schema means recording the expected structure and alerting when columns appear, disappear or change type.
Distribution is subtler. Here you look at the shape of the values: the percentage of nulls, the proportion between categories, the mean and spread of a numeric field, the number of distinct values. A country column that always had twenty values and drops to two, or an average price that triples from one day to the next, are signs that something changed at the source — even if volume and freshness are flawless. It is the difference between knowing the data arrived and knowing the data makes sense.
Lineage: from symptom to cause and impact
When an alert fires, two questions follow: what caused this and who is affected? Lineage answers both. By mapping how each table derives from others and which reports depend on each table, you can climb the chain to the origin of the problem and, at the same time, list the dashboards that should carry a "data under suspicion" warning.
Without lineage, every incident is an investigation from scratch, done in a hurry and under pressure. With lineage, root-cause analysis and impact analysis become almost immediate. Lineage is also what allows transparent communication: proactively warning those who use an affected report builds far more trust than letting the user discover the error.
How to start without buying a platform
You do not need to acquire a dedicated tool to take the first steps. Much of the value comes from simple queries over the metadata the warehouse already keeps. A freshness check can be as direct as comparing a table's maximum date with the current moment:
SELECT MAX(data_carga) AS ultima, CURRENT_TIMESTAMP AS agora FROM vendas;
A volume check compares the day's count with the recent average:
SELECT COUNT(*) FROM vendas WHERE data_carga = CURRENT_DATE;
From here, the path is incremental: schedule these queries, store the history of the results, define thresholds and send an alert when they are crossed. Start with the most critical tables — the ones the reports management sees depend on — and widen coverage as the value proves itself. Only when the number of tables and pipelines makes the manual effort unmanageable does it make sense to evaluate a dedicated platform with automatic anomaly detection.
Mini-case: a retailer that stopped learning about errors from its customers
Consider a retailer with a dozen stores and a handful of analysts. Sales and stock reports were fed by several nightly pipelines. About once a month a report would come up wrong — and, almost always, it was a store manager who reported it, which eroded trust in the whole data platform.
The team started by instrumenting only the five most important tables, with freshness and volume checks and a handful of distribution rules on critical fields, such as the percentage of nulls in the product identifier. Within three months, the mean time to detect a problem fell from more than a day to less than an hour, and most incidents began to be caught before reaching a report. The number of errors reported by business users dropped to almost zero.
The gain was not only technical. By proactively warning that a dashboard might be affected and was being fixed, the data team traded its image as a source of unpleasant surprises for that of a reliable service. Observability paid for itself not in lines of code, but in recovered trust.
Metrics to prove the value
Like any investment, observability should show results. Three metrics help tell the story. The mean time to detect measures how long it takes to discover a problem after it happens; the mean time to resolve measures how long it takes to fix it. Both should fall as coverage grows.
The third is data downtime: the period during which data was wrong, missing or partial, typically estimated as the number of incidents multiplied by the sum of the times to detect and to resolve. It is an honest measure of the risk the business was exposed to — and the most eloquent way to justify the effort to whoever decides the budget.
In practice
Data observability is neither a fad nor something that requires starting by buying software. It starts with a change of posture: assuming that data will fail in ways we cannot anticipate, and instrumenting the pipelines so that those failures announce themselves early instead of being discovered by the customer.
Choose the tables you depend on most, start by watching freshness and volume, add a few distribution rules on the fields that hurt most when they fail, and map the lineage that links each report to its sources. It is a modest investment to begin with and a clear return: fewer surprises, safer decisions and a data team the organisation trusts again.
