Methodology: how strategies are tested before they trade real money
A strategy trades real money only after it passes a fixed gate on paper trading: Brier score under 0.22, win rate at least 58%, Sharpe ratio at least 1.5, over a rolling 90 days and at least 30 resolved predictions, all at once. Live strategies are watched against the same gate and moved back to paper automatically if they fall behind. We publish no return figures.
What we measure, and why
- Brier score: the average of (forecast − outcome)² over resolved predictions. 0 is perfect; always saying 50% scores 0.25. It rewards forecasts that are both right and honest about their confidence. Try the Brier score calculator.
- Calibration: when the model says 70%, does the event happen about 70% of the time?
- Win rate and Sharpe ratio: measured on paper trades, after fees, to check the forecast turns into trades worth taking.
We don’t use return figures as evidence. A short run of profits says little, and a backtested equity curve is easy to fit to the past.
Calibration test: fast BTC contracts
The question: will BTC close the hour above a set price? With 5 to 30 minutes left, the model gives a probability, and we check it against what happened. The test covers 30 days, with 7 strike prices each hour.
| Time left | Brier score | Contracts |
|---|---|---|
| 5 min | 0.064 | 5,040 |
| 10 min | 0.092 | 5,040 |
| 20 min | 0.117 | 5,040 |
| 30 min | 0.140 | 5,040 |
Calibration, 10 minutes before the close: when the model said 12%, it happened 10% of the time; at 50%, 51%; at 88%, 92%.
Limits of this test
- Model only. No market prices, no orders, no fees. It shows the probabilities are calibrated, not that they beat the price on any venue.
- Synthetic strikes. Spot, and 0.1%, 0.2% and 0.4% above and below it. Short, near-spot contracts are easier to forecast, so don’t compare these scores with other markets.
- Correlated samples. Strikes in the same hour move together, so the effective sample is smaller than 5,040.
- Not tuned. The volatility multiplier is fixed at 1.0, not fitted to this data.
The live gate
To trade real money, a strategy must pass all five at once on paper trading:
| Measure | Required |
|---|---|
| Brier score | < 0.22 |
| Win rate | ≥ 58% |
| Sharpe ratio | ≥ 1.5 |
| Window | rolling 90 days |
| Sample | ≥ 30 resolved |
With fewer than 30 resolved predictions there is no score, so no promotion. The gate is written in code and tested; it isn’t adjusted after results come in.
Staying live
- Watch: if the rolling 30-day Brier score comes within 0.01 of the gate, position size is halved.
- Paper: past the gate, the strategy goes back to paper automatically. No real orders.
- Retire: still failing after 30 days, it is retired, with the reason written to the audit chain.
Sizing and exits
Each position starts at a quarter of the Kelly size for the model’s edge after fees. It is then cut to fit your daily loss, drawdown, exposure and open-position limits. The smallest size wins.
- Prediction-market strategy (paper): take profit at +22%, stop at −10%. Once up 12%, it exits if it gives back 4 points from the peak. Time stop at 1.5 hours, and it exits before the market resolves.
- Fast BTC contracts: no stop. Held to resolution, so the most a position can lose is its stake, sized as above.
What we don’t claim yet
We don’t publish a figure for beating prediction-market prices. On 30 held-out Polymarket markets, the calibrated model scored a slightly better Brier score than the market price overall, but only 6 of the 30 markets improved. That isn’t evidence of an edge. We’ll publish a number, with fees included, once 30 or more live predictions have resolved.
Questions
What returns does AI Trading Fleet make?
We don’t publish return figures. Strategies are judged on forecast accuracy and paper-trading results against a fixed gate, and trading can lose money.
What is a Brier score?
The average of (forecast − outcome)² over resolved predictions, with outcomes scored 1 or 0. 0 is perfect, 0.25 is what always saying 50% scores, and lower is better.
When does a strategy trade real money?
Only after it passes, on paper trading and all at once: Brier score under 0.22, win rate at least 58%, Sharpe ratio at least 1.5, over a rolling 90 days with at least 30 resolved predictions.
What happens if a live strategy gets worse?
Its position size is halved when it nears the gate, it goes back to paper automatically once it fails the gate, and it is retired if it still fails after 30 days.
Reserve Early Access for $10 in crypto: 50% off your first 12 months, deposit credited to your first payment.
Trading involves risk of loss. Nothing here is investment advice. Read the risk disclosure.