Brier score calculator
- Always saying 50%
- —
- Always saying the base rate
- —
- Skill score vs base rate
- —
- Log loss
- —
- Reliability (lower is better)
- —
- Resolution (higher is better)
- —
- Market on the same questions
- —
Example values. Change any field. Everything is calculated in your browser and nothing is sent anywhere.
The Brier score is the average of (forecast − outcome)² over resolved predictions, with outcomes scored 1 or 0. 0 is perfect, 0.25 is what always saying 50% scores, and lower is better. AI Trading Fleet only lets a strategy trade real money with a Brier score under 0.22 on at least 30 resolved predictions.
How is the Brier score calculated?
Brier = (1/N) × Σ (forecast − outcome)² Skill score = 1 − Brier ÷ Brier of always forecasting the base rate Log loss = −(1/N) × Σ [ y·ln(p) + (1 − y)·ln(1 − p) ] outcome y = 1 if the event happened, 0 if not
A forecast of 70% on an event that happens scores (0.7 − 1)² = 0.09. The same forecast on an event that doesn't scores 0.49. Confident misses cost the most.
What is a good Brier score?
Always saying 50% scores 0.25, so anything above that is worse than a coin flip. Below that, "good" depends on how predictable the questions are, so compare on the same questions. For trading, the benchmark is the market: if the market price scores better than your forecasts on the same events, there is no edge to trade.
The skill score measures against always forecasting the base rate. Above 0 means your forecasts add information; 0 or below means they don't.
Calibration, reliability and resolution
The Brier score splits into three parts. Reliability is how far your stated probabilities sit from what actually happened: when you say 70%, does it happen about 70% of the time? Resolution is how far your forecasts move away from the base rate when they should. Uncertainty is how hard the questions are, and is out of your hands. Brier ≈ reliability − resolution + uncertainty; with binned forecasts the parts may not add up exactly.
Mistakes that make a Brier score look better than it is
- Scoring on the data you tuned on. Only forecasts made before the model saw the outcomes count.
- Lookahead. A forecast timestamped after information that settles the question is not a forecast.
- Tiny samples. 20 or 30 forecasts can't separate skill from luck. Watch the score over a rolling window.
- Picking the subset. Scoring only the trades a filter chose, and comparing with the market on those same trades, can flatter the model. Report the full set too.
How AI Trading Fleet uses the Brier score
To go live, a strategy has to clear all five at once on its paper account: a Brier score under 0.22, a win rate of at least 58%, a Sharpe ratio of at least 1.5, over a rolling 90-day window, with at least 30 resolved predictions.
It keeps being scored after it goes live. If its Brier score drifts to within 0.01 of the gate, its order size is halved. If it crosses the gate, it goes back to paper and sends no real orders. If it keeps failing on paper for 30 days, it is retired. Putting it back on real money needs a person to sign off.
Questions
What is a good Brier score?
Lower is better. 0 is perfect and 0.25 is what always forecasting 50% scores. What counts as good depends on the questions, so compare against a benchmark on the same questions, such as the market price.
What is the difference between Brier score and log loss?
Both score probability forecasts. Log loss punishes confident misses much harder and is infinite for a 0% or 100% forecast that turns out wrong. The Brier score is bounded between 0 and 1.
How many forecasts do I need for a Brier score to mean anything?
More than most people use. AI Trading Fleet requires at least 30 resolved predictions in a rolling 90-day window before a strategy is judged, and a few hundred give a much steadier number.
Can I compare my forecasts with Polymarket or Kalshi prices?
Yes. Add the market price at the time of your forecast as a third column. The calculator scores the market on the same questions so you can see whether you beat it.
Reserve Early Access for $10 in crypto: 50% off your first 12 months, deposit credited to your first payment.
Trading involves risk of loss. Nothing here is investment advice. Read the risk disclosure.