Ember-0
16 tasks, from retrieving an out-of-reach plush and serving cereal to using tongs, writing "HI", unfolding a T-shirt and tying a knot.
As of November 20, 2026, GPT-6.2 Astra leads with 41.2, followed by GPT-6 Astra (19.6) and GPT-6.1 Sol (19.3). Official leaderboard ↗ · Paper ↗
Leaderboard
- Tasks
- 16
- Attempts / task
- 30
- Median cost / task
- $4.57
- Median time / task
- 21 min 27 s
Overall vs speed
Published time per task attempt
Overall vs cost
Published cost per task attempt
Success rate (%), the mean of the 16 tasks with equal weight, 30 trials per task before XDOF's exclusions.
GPT-6.2 Astra41.296.763.379.3$7.0117 min 28 s2
GPT-6 Astra19.663.352.018.5$10.0920 min 26 s3
GPT-6.1 Sol19.382.875.90.00$2.1119 min 23 s4
Claude Fable 5.219.266.770.043.3$7.2122 min 04 s5
Claude Opus 5.511.655.255.20.00$5.0121 min 27 s6
Claude Fable 5.19.186.233.30.00$7.1621 min 26 s7
Gemini 4 Argon7.963.343.33.3$1.9226 min 00 s8
Muse Wonder 1.04.450.017.20.00$4.5728 min 49 s9
Gemini Robotics ER 2 Preview0.000.000.000.00$3.3425 min 00 s10
Grok 4.70.000.000.000.00$1.0929 min 32 sTasks
16 tasks.
Methodology
- Scoring
- Pass/fail + checkpoints
- Time cap / attempt
- 30 min 00 s
- Control rate
- —
- Models in index
- 10 of 10
Scoring. Annotators mark whether each trial meets the task's written success criterion, and which checkpoints it reached.
Who runs it. XDOF runs every trial on its own stations and annotates success and checkpoints.
Robots. I2RT YAM (two-arm), six stations.
Note. Coding agents only, in XDOF's harness at high reasoning effort. 30 trials per task; XDOF excludes invalidated takes, leaving 449 to 480 scored runs per model. XDOF sells evaluations and robot training data to AI labs.
Time per attempt. Published or measured for 10 of 10 models.
Cost per attempt. Published for 10 of 10 models.