The Brier Score, Explained Properly
The standard way to score a probabilistic forecast, what a given number actually means, and why it rewards honesty rather than confidence.
If you want to know whether your forecasts are any good, you need a number. The Brier score is the standard one, and it is simple enough to compute in your head.
The formula
Take the probability you gave, as a decimal. Subtract the outcome, which is 1 if the thing happened and 0 if it did not. Square the result.
Say 80% on something that happens: (0.8 − 1)² = 0.04. Say 80% on something that does not: (0.8 − 0)² = 0.64.
Average across all your forecasts. Lower is better. Zero is perfect. A quarter is what you get by saying 50% every time.
What a number means
0.25 is the score for saying 50% to everything. Anything worse than this means you are actively worse than uninformative.
Around 0.18 to 0.20 is typical for a thoughtful amateur on a mixed set of questions.
0.10 to 0.15 is good, on a genuinely difficult question set.
Below 0.10 usually means the questions were easy rather than that the forecaster was brilliant. Scores are not comparable across question sets, and this is the single most important caveat.
That last point is worth repeating. A Brier score is meaningless in isolation. It is only interpretable against another forecaster, or a market, or a base-rate benchmark, on exactly the same questions.
Why it rewards honesty
The Brier score is a proper scoring rule. Your expected score is minimised by reporting exactly what you believe.
Overstate your confidence and the occasional miss costs you disproportionately, because the error is squared. Hedge toward 50% and you give up the credit you would earn for being right. Neither distortion pays.
This is not a coincidence; proper scoring rules are designed with this property, and it is why they are used anywhere forecasts are seriously evaluated. Log scoring is the other standard choice and it punishes confident errors far more harshly.
Decomposing it
A Brier score splits into parts that are individually informative.
Calibration. Do your 70% forecasts happen 70% of the time. Calibration curves.
Resolution. How far from the base rate you are willing to go, and whether you are right when you go there. A perfectly calibrated forecaster who never leaves the base rate has good calibration and zero resolution.
Uncertainty. A property of the questions, not of you. It is why scores across different question sets cannot be compared.
Most people who improve find that calibration improves first and resolution takes much longer, because resolution requires actually knowing something.
Benchmarking against the market
The most useful application. For each question, score your forecast and score the market price at the same moment. Compare the averages.
If the market beats you, you are reading the market and adding noise. That is the normal result and finding it out early saves a great deal of money. If you beat the market consistently across a meaningful number of questions, you have something — though see luck versus skill in small samples before celebrating, because the number of questions required is larger than people expect.
Practical use
Score everything you record. It takes seconds per forecast and it turns a journal from a diary into an instrument. Keeping a forecast journal.
Do not chase the score by picking easy questions. A brilliant score on questions whose answers were obvious measures nothing.
Next
Log scoring, and why it is harsher.
Educational content only. Nothing here is financial advice, a solicitation, or an offer to trade. Prediction markets can move against you and a position can lose its entire value. Where this guide describes a difference between venues it describes a theoretical cross-market edge in observed prices, which is not a profit: fees, spreads, capital lockup, transfer time and differing settlement rules all come out of it first.
Keep reading
Forecasting and Calibration · 3 min readWhere the Wisdom of Crowds Runs OutAggregated opinion beats most individuals most of the time. The conditions it depends on are specific, and prediction markets sometimes fail to meet them.
Forecasting and Calibration · 3 min readLog Scoring, and Why Confident Errors Cost So MuchThe other proper scoring rule. It is unbounded, it punishes confident mistakes severely, and that severity is the argument for using it.
Forecasting and Calibration · 3 min readKeeping a Forecast Journal That Is Actually UsefulMost forecasting improvement comes from records, not from cleverness. What to write down, what to leave out, and how to review it without flattering yourself.
What Markets Price · 3 min readAward and Culture MarketsPrizes, ceremonies and cultural milestones. Small, thin, entertaining, and a surprisingly good place to practise forecasting against a clean outcome.