Log Scoring, and Why Confident Errors Cost So Much
The other proper scoring rule. It is unbounded, it punishes confident mistakes severely, and that severity is the argument for using it.
The Brier score is easy to compute and easy to explain. The log score is neither, and it is what most serious evaluation uses anyway. Here is why.
The rule
Take the probability you assigned to what actually happened. Take its natural logarithm. Negate it.
Say 80% on something that happens: −ln(0.8) = 0.22. Say 80% on something that does not: −ln(0.2) = 1.61.
Lower is better. Zero is perfect and only achievable by assigning 100% to something that happens.
The important property: it is unbounded
Assign 1% to something that then happens and your score for that question is −ln(0.01) = 4.61. Assign 0.1% and it is 6.91. Assign zero and it is infinite.
Under Brier, the worst possible single-question score is 1. Under log scoring there is no worst. One confidently wrong forecast can dominate a year of good ones.
Why that is the right design
Because in most real settings, confident errors genuinely are catastrophically worse than hedged ones. A forecaster who says 99% and is wrong has not made a slightly larger mistake than one who said 90%; they have made a qualitatively different one. They claimed there was almost no chance, and there was.
Brier's squared error compresses that difference. Log scoring does not. If you want a metric that hunts down exactly the failure mode most forecasters actually have — overconfidence at the extremes — this is the one.
It is also the reason the rule says: never say zero or one. Not as a stylistic preference, but because the score for being wrong is infinite. Prices near 1c and 99c is the market version of the same point.
The information interpretation
The log score is the number of nats of surprise in the outcome given your forecast. Averaged over many forecasts it is cross-entropy: how many bits, on average, it costs to encode what happened using your beliefs as the code.
A perfect forecaster costs zero. A forecaster with no information costs the entropy of the outcomes themselves. The difference is, literally, how much information you had.
This is why the same rule appears as the loss function in classification models. It is not a coincidence or an analogy; it is the same quantity.
Which to use
Brier for explaining, for quick mental arithmetic, and when one bad forecast dominating the average would be misleading.
Log for serious evaluation, when overconfidence is the failure mode you care about, and when you want the information-theoretic interpretation.
Both on the same journal, which costs nothing extra and occasionally disagrees in a way that tells you something. A large gap between your Brier ranking and your log ranking means your errors are concentrated in a few confident misses.
Applied to markets
Score market prices with the same rule and you get a direct measure of how much information a venue's prices carry. Do it by category and you learn where a venue is informative and where it is guessing.
Do it by liquidity bucket and you will find, reliably, that liquid markets score much better than thin ones. That is the quantitative version of the advice to weight a price by its volume. Where crowd wisdom runs out.
The rule that falls out
Never assign zero. Never assign one. Not even to things that seem certain, because the class of things that seemed certain and were not is not empty.
Cap yourself at something like 1% and 99% unless you have a genuinely mechanical reason to go further. The expected cost of that cap is tiny; the expected cost of not having it is unbounded.
Next
Updating on news — moving your number for the right reasons.
Educational content only. Nothing here is financial advice, a solicitation, or an offer to trade. Prediction markets can move against you and a position can lose its entire value. Where this guide describes a difference between venues it describes a theoretical cross-market edge in observed prices, which is not a profit: fees, spreads, capital lockup, transfer time and differing settlement rules all come out of it first.
Keep reading
Forecasting and Calibration · 3 min readThe Brier Score, Explained ProperlyThe standard way to score a probabilistic forecast, what a given number actually means, and why it rewards honesty rather than confidence.
Forecasting and Calibration · 3 min readUpdating on News Without OverreactingNew information should move your forecast by an amount proportional to how surprising it is. Most people move too far on noise and not far enough on signal.
Forecasting and Calibration · 3 min readWhere the Wisdom of Crowds Runs OutAggregated opinion beats most individuals most of the time. The conditions it depends on are specific, and prediction markets sometimes fail to meet them.
Order Books and Liquidity · 3 min readExit Early or Hold to SettlementTaking a profit before resolution frees capital and pays a spread. Holding pays no spread and ties up capital. The arithmetic that decides it.