Online Tool Store
· 4 min read
How to Compare Two Versions of a Dataset
Heshan Fernando
Co-founder & COO
The monthly report comes out and one number looks wrong. Not impossibly wrong — just lower than expected, in a way that could be seasonal.
It takes three days to establish that a source field stopped populating on the 8th. The comparison that would have shown it took two minutes and nobody ran it.
The statistics that catch problems
Comparing two snapshots column by column, four measures do most of the work.
Null rate is the most informative and the most ignored. A column going from 0.2% null to 11% null is not a business change. It is a join that stopped matching, a renamed source field, or an upstream system that started omitting something.
Distinct value count catches a different failure. A category column dropping from 40 distinct values to 6 means most categories stopped arriving. A column going to exactly 1 distinct value means it stopped varying at all.
Mean and median together show whether a shift is broad or driven by outliers. A mean that moves while the median holds is a few extreme values; both moving is the whole distribution.
Row count seems obvious and is worth stating, because a 3% drop is easy to miss and a pipeline that silently loses rows is a real failure mode.
| Change | Usually means |
|---|---|
| Null rate up sharply | Broken join or renamed field |
| Distinct count collapses | Filter applied upstream |
| Mean moves, median holds | A few outliers |
| Both move together | Genuine distribution shift |
| Row count down slightly | Silent pipeline loss |
Structural changes matter as much as values
Columns present in one snapshot and absent from the other are a finding in themselves.
A new column is usually harmless and worth knowing about. A missing column means something downstream is about to fail, or already has and is defaulting silently.
A renamed column shows as both at once — one disappeared, one appeared — which is a pattern worth recognising because it is the most common schema change nobody announces.
Thresholds, and when to worry
Setting a threshold decides how much noise you see. Everything changes a little between snapshots.
A useful starting point is to flag anything moving more than 10%, then tune. Some columns are naturally volatile and will always flag; excluding them explicitly is better than raising the global threshold until nothing shows.
What matters more than the threshold is which measure moved. A mean drifting 8% is often real. A null rate moving 8 percentage points almost never is.
Compare like periods
A comparison between two snapshots assumes they are comparable, and frequently they are not.
Different lengths. A 28-day period against a 31-day one differs by roughly 10% on any volume measure before anything real has changed.
Different points in a cycle. A month ending mid-billing-cycle against one ending at the close contains a different mix of transactions.
Seasonality. December against November is a comparison of two seasons, not two states of a system.
The fix is either comparing equivalent periods — this month against the same month last year — or normalising to a daily rate before comparing. Doing neither produces differences that are real, explainable, and nothing to do with data quality.
Common mistakes to avoid
- Comparing totals rather than distributions, which hides everything except the largest changes.
- Ignoring null rate because the column still has data in it.
- Setting one threshold across all columns and then raising it until the noise stops.
- Comparing snapshots from different points in a monthly cycle, where the difference is the cycle rather than a problem.
- Running the comparison once after a failure rather than on a schedule, which is the version that prevents the next one.
How to do it with Column Statistics Comparator
The Column Statistics Comparator diffs two datasets column by column in the browser.
- Paste both snapshots with matching headers — last month and this month, or before and after a pipeline change.
- Set the threshold above which a difference is flagged.
- Read null rate and cardinality before the means.
- Check the structural differences too — a column present in only one snapshot is its own finding.
Other data tools that keep files local are in the tools directory.
Frequently asked questions
Which changes matter most?
Null rate and distinct value count. A mean shifting a few percent is often genuine; a null rate jumping from near zero is almost always a broken join or a renamed source field.
Is this the same as data drift monitoring?
It is the manual version. Continuous monitoring runs these comparisons on a schedule and alerts on thresholds, which is what catches a problem on the day rather than at month end.
What if the two files have different columns?
That is reported separately and is itself a finding. A column appearing in one snapshot and not the other usually means a schema change nobody announced, and a rename shows as one disappearing and one appearing.
Final thought
Compare the null rates first. A column that still has data in it can have stopped working, and the mean will not tell you.