Detecting Stale Data Before It Reaches Anyone
A pipeline that keeps running while its source stops updating is worse than one that fails loudly. How to measure freshness and what to do when it slips.
The worst failure in a data pipeline is not a crash. A crash is obvious. The worst failure is everything continuing to run while the numbers quietly stop being current, because nothing is broken and nobody notices.
The three ages
Every value should carry three timestamps, and confusing them is the root of most staleness problems.
Source age. How long since the venue itself updated this value. The one that determines whether a number is current.
Ingest age. How long since your collector successfully fetched it. Determines whether your pipeline is healthy.
Publish age. How long since it reached whatever surface a reader sees. Determines whether your cache or build step is lying.
A number can be fresh by two of these and stale by the third, and each combination means something different.
What to measure
Per source. Time since the last successful fetch. Time since the last fetch that produced a change.
Per market. Age of the quote. Age of the last trade. These differ and both matter. Mid price versus last trade.
Per derived output. How old the oldest input to a computed figure is. A cross-venue gap is exactly as fresh as its staler side, and that is the number to attach to it.
The distinction that catches people
"No change" and "no data" look identical unless you record them differently.
A market that has not moved for six hours and a collector that has been failing for six hours produce the same series. Write a heartbeat row on every successful fetch, even when nothing changed, and record failures explicitly as rows. Then the two are trivially distinguishable, forever.
Thresholds worth setting
Per source freshness. If a source has produced no successful fetch in some multiple of its expected interval, that is an alarm. The multiple should reflect the polling tier, not a single global number.
Per market quote age. Beyond a threshold, exclude that market from comparisons rather than showing a stale number as if it were current.
Cross-venue timestamp skew. If the two sides of a comparison are more than a short window apart, do not report a gap. The difference you would be reporting includes the skew. Stale quotes and timing.
Derived-output age. If a page is serving figures computed from inputs older than a threshold, say so on the page.
What to do when it slips
Degrade visibly, not silently. Show the age. A number labelled "as of four hours ago" is useful; the same number presented as current is misinformation.
Exclude rather than estimate. Dropping a stale market from a comparison table costs a reader nothing. Including it costs them trust in every other row.
Alert on the condition, not on a proxy for it. An alarm on "no rows written" misses a collector that is writing failures successfully. Alert on source freshness directly.
Make the alarm boring. An alert that fires constantly gets ignored, and then the real one is ignored too.
Why this matters more here than elsewhere
Because the central product of a cross-market comparison is a difference between two numbers observed at two moments. If either moment is wrong, the difference is wrong, and it will be wrong in the direction of looking more interesting.
Staleness does not produce random errors. It produces large apparent opportunities. That asymmetry is why freshness monitoring is not an operational nicety here; it is part of the correctness of the output. Cross-market mistakes.
Next
Educational content only. Nothing here is financial advice, a solicitation, or an offer to trade. Prediction markets can move against you and a position can lose its entire value. Where this guide describes a difference between venues it describes a theoretical cross-market edge in observed prices, which is not a profit: fees, spreads, capital lockup, transfer time and differing settlement rules all come out of it first.
Keep reading
Data and Automation · 3 min readRate Limits, Polling, and Not Being a NuisanceA polling schedule that matches where the information actually is will use a fraction of the requests and produce better data than polling everything.
Data and Automation · 3 min readNormalising Market Data Across VenuesTwo venues describe the same idea with different shapes. A documented intermediate representation is the difference between a comparison and a guess.
Data and Automation · 3 min readStoring Market Time Series Without Drowning in DataMost of what a naive collector records says nothing happened. Sampling, change-only writes and sensible retention keep a useful history small.
Order Books and Liquidity · 3 min readReading an Order Book ProperlyA book is more informative than a price. How to weight it, what a lopsided book means, and the patterns that tell you a quote is not to be trusted.