Luck, Skill, and How Many Forecasts You Actually Need
Twenty correct calls prove very little. A short look at how much evidence it takes to distinguish a real edge from an ordinary run of good luck.
Someone goes fifteen for twenty on their forecasts and concludes they have an edge. The arithmetic says otherwise, and the arithmetic is worth internalising before you size anything.
The coin-flip baseline
If you make twenty independent forecasts at 50% and simply guess, you expect ten right. Getting fifteen or more happens by chance about 2% of the time.
That sounds impressive until you notice how many people are making forecasts. Across a large population, plenty of people go fifteen for twenty on pure luck, and every one of them has a compelling story about their method.
Why forecasting is worse than coin flips
Three complications, all working against you.
Questions are not equally hard. A set full of 90% questions will produce a high hit rate from anyone. Hit rate without difficulty adjustment is not a measure of anything, which is why proper scoring rules exist. The Brier score.
Outcomes are correlated. Ten forecasts on related questions are not ten independent observations. One underlying event being right makes all of them right together. Your effective sample is much smaller than your count.
Selection. You remember the run you are on. You are looking at your record because it is good.
How much you need
Roughly, to distinguish a genuine small edge from noise you need hundreds of independent forecasts. To distinguish a small edge over a market benchmark, more.
This is the same arithmetic that makes evaluating fund managers hard, and it has the same conclusion: the sample sizes required are larger than the samples people have.
What to do about it instead
Score against a benchmark, not against zero. Compare to the market price at forecast time. This removes question difficulty from the comparison and dramatically improves the signal per observation. Log scoring.
Use proper scoring rules. They extract more information per forecast than a hit rate, because they use the probability you assigned rather than just whether you were on the right side.
Count independent events, not forecasts. Group correlated questions and treat the group as one observation.
Look at calibration, not just average score. A calibration curve is informative with less data than a mean-score comparison, because deviations show up as a systematic pattern rather than as a single noisy number. Calibration curves.
The practical consequences
Do not size up after a good run. The run is the least reliable evidence you will encounter and it arrives exactly when you are most inclined to trust it. Position sizing.
Do not abandon a method after a bad run either. Same arithmetic, opposite direction.
Be suspicious of anyone's track record, including your own. Especially when the sample is short and the questions were chosen by the person showing you the record.
Expect drawdowns that are indistinguishable from being wrong. With a genuine small edge, long losing stretches are normal. You cannot tell them from having no edge by looking at the results alone, which is why the reasoning has to be checkable independently. Variance and drawdown.
The honest position
Most people who believe they have a forecasting edge have a sample too small to support the belief. That includes people who are right.
The correct response is not paralysis. It is to keep records, score properly against a benchmark, size as though you might be wrong, and let the evidence accumulate at the speed it accumulates.
Next
Aggregating forecasts — combining several estimates into a better one.
Educational content only. Nothing here is financial advice, a solicitation, or an offer to trade. Prediction markets can move against you and a position can lose its entire value. Where this guide describes a difference between venues it describes a theoretical cross-market edge in observed prices, which is not a profit: fees, spreads, capital lockup, transfer time and differing settlement rules all come out of it first.
Keep reading
Forecasting and Calibration · 3 min readThe Habits That Actually Correlate With AccuracyResearch on forecasting tournaments found consistent behavioural differences in the best performers. Most of them are habits rather than talents.
Forecasting and Calibration · 3 min readCombining Several Forecasts Into a Better OneAveraging estimates usually beats picking the best one. Which average to use, why extremising helps, and how a market price is itself an aggregate.
Forecasting and Calibration · 3 min readUpdating on News Without OverreactingNew information should move your forecast by an amount proportional to how surprising it is. Most people move too far on noise and not far enough on signal.
Polymarket · 3 min readStablecoin Collateral and What Self-Custody Means HerePositions are backed by a dollar-pegged stablecoin held in a contract. What that protects you from, what it does not, and the security rules that follow.