Haiku 5.5
3 runs · 445 of 450 tasks so far- Mean reward
- 0.48runs 0.48–0.49
- Pass rate
- 2.5%runs 2.2–2.7%
- Median
- 0.46
- Per trial
- $0.074
Share of trials by reward, bands of 0.1
Trial results · updated as the trials run
How the models score.
Each task asks a model to reconstruct a famous body of measured natural-science knowledge, item by item, from what it has learned: no internet access, one hour, closed answer options. These are the model-level results so far.
Updated 13 October 2026, 01:47 UTC
450tasks in the benchmark
13fields of science
6models on trial
2,500graded trials
01 / The models
Far from full marks.
A reward of 1 means every item was reconstructed exactly. The best model so far, GPT-6.1 Sol, averages 0.66. A trial passes when it scores above 0.95. Haiku 5.5 ran every task three times; its figures pool the three runs, with the range of the runs beneath.
GPT-6.1 Sol
445 of 450 tasks so far- Mean reward
- 0.66
- Pass rate
- 9.4%
- Median
- 0.65
- Per trial
- $0.68
Share of trials by reward, bands of 0.1
GPT-6 Luna
364 of 450 tasks so far- Mean reward
- 0.46
- Pass rate
- 2.5%
- Median
- 0.42
- Per trial
- $0.019
Share of trials by reward, bands of 0.1
GPT-5.6 Sol
349 of 450 tasks so far- Mean reward
- 0.60
- Pass rate
- 5.2%
- Median
- 0.58
- Per trial
- $1.84
Share of trials by reward, bands of 0.1
Also under way: Sonnet 5.5 (6 graded trials), Opus 5.5 (1 graded trial). A model joins the charts once it has 30 graded trials.
02 / Worked examples
See what the models drew.
Two tasks shown in full: the question, the reference, and every model’s answer laid over it, item by item.
Earth structure
The kinds of plate boundary
Every 1.8° cell along Earth’s plate boundaries, labelled with its boundary class: spreading ridge, transform fault, subduction zone and four more.
- Haiku 5.5 0.54
- GPT-6.1 Sol 0.65
- GPT-6 Luna 0.52
- GPT-5.6 Sol 0.53
The Milky Way in Gaia star counts
How many stars brighter than G = 19 fill each of 3,072 patches of sky in ESA’s Gaia DR3 catalogue: the Galaxy, drawn from what a model remembers.
- Haiku 5.5 0.61
- GPT-6.1 Sol 0.88
- GPT-5.6 Sol 0.86
03 / Score and cost
What a better score costs.
Each point is one model: its cost per trial at public list prices, against how often it passes and how well it scores on average. Haiku 5.5’s bars span its lowest to its highest run, in score and in cost. Hover a point for its numbers.
Pass rate against cost
Share of trials scoring above 0.95.
Mean reward against cost
Average score over all graded trials.
| Model | Trials | Mean reward | Pass rate | Passes | Per trial | Total |
|---|---|---|---|---|---|---|
| Haiku 5.5 | 1335 | 0.4840.483–0.487 | 2.5%2.2%–2.7% | 33 | $0.074 | $99 |
| GPT-6.1 Sol | 445 | 0.656 | 9.4% | 42 | $0.68 | $301 |
| GPT-6 Luna | 364 | 0.461 | 2.5% | 9 | $0.019 | $6.82 |
| GPT-5.6 Sol | 349 | 0.601 | 5.2% | 18 | $1.84 | $641 |
Costs are estimates from each trial’s token counts at the providers’ public list prices, not amounts billed. Trials that failed for infrastructure reasons, before the model could answer, are excluded and run again.
04 / By field
From the atom to the observable universe.
Mean reward per field. Deeper gold is higher. Hover a cell for the number of trials behind it.
| Field | Haiku 5.5 | GPT-6.1 Sol | GPT-6 Luna | GPT-5.6 Sol |
|---|---|---|---|---|
| Atoms & molecules21 tasks | 0.55 | 0.73 | 0.50 | 0.62 |
| Cosmology16 tasks | 0.40 | 0.51 | 0.43 | 0.51 |
| Earth's deep past45 tasks | 0.44 | 0.60 | 0.45 | 0.55 |
| Earth structure45 tasks | 0.42 | 0.60 | 0.40 | 0.50 |
| Galaxies & Milky Way43 tasks | 0.45 | 0.61 | 0.42 | 0.56 |
| High-energy sky29 tasks | 0.47 | 0.67 | 0.52 | 0.65 |
| Matter38 tasks | 0.68 | 0.79 | 0.56 | 0.73 |
| Molecules of life54 tasks | 0.53 | 0.63 | 0.48 | 0.58 |
| Nuclear33 tasks | 0.61 | 0.82 | 0.59 | 0.80 |
| Particle27 tasks | 0.57 | 0.82 | 0.55 | 0.74 |
| Planets & small bodies54 tasks | 0.35 | 0.52 | 0.34 | 0.47 |
| Stars & Sun39 tasks | 0.41 | 0.65 | 0.40 | 0.58 |
| Organisms6 tasks | 0.60 | 0.70 | 0.58 | 0.56 |
05 / How it is run
The same exam, for every model.
Blind and offline
Every model gets the same task in the same isolated container: no internet, no web search, one CPU, one hour and at most 100 turns. Models run at their highest reasoning setting.
Graded against the record
Each item takes one label from a closed set. An exact answer scores 1, an adjacent bin 0.5, anything else 0. The reward is the plain mean over the items, with no rescaling.
Published in stages
The worked examples are shown in full, question and answers. The other tasks are not published yet; their results appear here only as model-level statistics.