Backtesting a Market Signal Honestly

Build a backtest of a cross-market strategy using last-trade prices and no cost model, and you will find an enormous edge. It will be entirely artificial. Here is where it comes from and how to remove it.

Error one: using prices nobody could trade

Last trade is a historical fact, not an available price. A backtest that buys at the last trade is assuming a counterparty who was not there. Mid price versus last trade.

Fix: simulate against the side you would actually cross. Buy at the offer, sell at the bid. If you did not record quotes, you cannot do this, which means you cannot backtest execution at all — and you should say so rather than substituting mids.

Error two: ignoring size

A signal firing on a market with forty contracts at the top of book cannot be traded in size. A backtest that assumes unlimited fills at the quote has invented liquidity.

Fix: cap simulated size at what the recorded book supported, and walk the book for anything larger. Depth and slippage.

Error three: survivorship

If your dataset only contains markets that resolved cleanly, you have excluded the voided ones, the ambiguous ones and the ones that never got liquid. Those are disproportionately the bad outcomes.

Fix: record every market from listing, including the ones that go nowhere.

Error four: lookahead

Using the final settled question text, or a category assignment made after the fact, or a matching decision that used information from after the trade. This is subtle and pervasive in cross-venue work, where the pairing itself is often made using knowledge of how both markets resolved.

Fix: every input to a decision at time T must have been observable at time T. Store data as it appeared, not as it was later corrected.

Error five: no cost model

Spread, fees, slippage, capital cost. On strategies whose whole edge is a few cents, these are not an adjustment — they are the entire result. How fees eat an edge and capital lockup.

Fix: model all four explicitly. If the strategy only works with zero costs, it does not work.

Error six: overfitting

With a few hundred resolved markets and enough parameters, you can fit anything. Thresholds tuned on the same data they are tested on are not evidence.

Fix: hold out time periods, not random samples. Prefer fewer parameters. Be extremely suspicious of any rule with a threshold that happens to sit just where the data is thin.

Error seven: too few independent events

A hundred markets on twenty underlying events is twenty observations, not a hundred. Correlated outcomes inflate apparent significance dramatically. Luck versus skill in small samples.

Fix: count independent events. Report that number, not the trade count.

What an honest backtest looks like

It reports the number of independent events. It uses executable prices with a depth cap. It includes a full cost model. It shows results out of sample. It states what it could not measure.

And it usually shows a much smaller edge than the first version did — often none at all. That is the backtest working.

The order of operations

Record first. Run a collector for a month with honest timestamps before you model anything. Then look. Then, if something survives, paper trade it forward before risking anything.

Almost everyone reverses this, models first, finds an edge made of artefacts, and learns the same lesson more expensively.

Next

Public market APIs — where the data comes from.