Rate Limits, Polling, and Not Being a Nuisance
A polling schedule that matches where the information actually is will use a fraction of the requests and produce better data than polling everything.
The naive approach is to poll everything as often as the limit allows. It produces mostly duplicate data, consumes headroom you will want later, and is inconsiderate. A little structure does far better.
Tier your polling
Hot. A small watchlist of liquid, active markets. Every thirty seconds or so.
Warm. Markets with recent activity but not currently interesting. Every few minutes.
Cold. Everything else. Hourly or less.
Static. Market metadata, resolution criteria, close times. Daily, or on change.
Promote and demote automatically based on recent trading. A market that starts moving should be hot within one cycle; one that has gone quiet should fall back.
Poll where the information is
Information density is not uniform in time.
Around scheduled events, poll hard for a short window and normally otherwise. You know when releases are; schedule against them. Macro release markets.
Overnight and at weekends, most markets do nothing. Back off.
When a market is repeatedly unchanged, increase the interval. When it changes, decrease it.
Back off properly
On any error, and especially on a rate-limit response, wait and retry with exponentially increasing delay plus a random component. Never retry immediately in a loop.
The random component matters if you run several workers: without it they will synchronise and hit the limit together, repeatedly.
Respect any retry-after header the service sends. It is telling you exactly what to do.
Budget your requests
Work out your allowance, subtract a safety margin, and allocate what remains across tiers deliberately. A hard cap in the client is worth having so a bug cannot consume the whole budget in a minute.
Leave headroom. You will want burst capacity around an event, and having spent it on polling dormant markets at three in the morning is an avoidable regret.
Conditional requests and caching
Where the service supports validators, use them: an unchanged response costs far less on both ends. Cache anything static locally rather than re-fetching it.
Distinguish "no data" from "no change"
A collector that is down looks exactly like a market that is not moving, unless you record the difference. Write a heartbeat row when you successfully fetch and find nothing changed, and record failures as rows of their own.
Without this, a gap in your data is ambiguous forever. Detecting stale data.
Alert on the pipeline, not just the prices
The failure that matters most is silent: a source stops updating, the collector keeps running, and every downstream number quietly ages. That condition needs its own alarm, separate from anything about price movement. Alerting on price moves.
Be a good citizen
Identify your client honestly. Do not attempt to route around rate limits. Do not scrape where an endpoint exists. Read the current terms.
Beyond being the right thing, it is also the pragmatic thing: the alternative is being blocked, and then you have no data at all.
Next
Detecting stale data before it reaches anyone.
Educational content only. Nothing here is financial advice, a solicitation, or an offer to trade. Prediction markets can move against you and a position can lose its entire value. Where this guide describes a difference between venues it describes a theoretical cross-market edge in observed prices, which is not a profit: fees, spreads, capital lockup, transfer time and differing settlement rules all come out of it first.
Keep reading
Data and Automation · 3 min readStoring Market Time Series Without Drowning in DataMost of what a naive collector records says nothing happened. Sampling, change-only writes and sensible retention keep a useful history small.
Data and Automation · 3 min readDetecting Stale Data Before It Reaches AnyoneA pipeline that keeps running while its source stops updating is worse than one that fails loudly. How to measure freshness and what to do when it slips.
Data and Automation · 3 min readHistorical Price Series and Their PeculiaritiesEvent contract history is not like equity history. Fixed bounds, a settlement point, irregular trading and a survivorship problem built into the data.
Risk and Costs · 3 min readThe Limits of Any Measured EdgeA measured price difference is a measurement. The distance between that and a return is made of costs, capacity, decay and the chance the measurement is wrong.