Benchmarks/Ember-0

Ember-0

Real robotsPass/fail + checkpointsReleased Oct 2026Index weight 0.81
XDOF EvaluationsMaintainer ↗

16 tasks, from retrieving an out-of-reach plush and serving cereal to using tongs, writing "HI", unfolding a T-shirt and tying a knot.

As of November 20, 2026, GPT-6.2 Astra leads with 41.2, followed by GPT-6 Astra (19.6) and GPT-6.1 Sol (19.3). Official leaderboard ↗ · Paper ↗

Leaderboard

Tasks
16
Attempts / task
30
Median cost / task
$4.57
Median time / task
21 min 27 s

Overall vs speed

Published time per task attempt

Faster, higher scoring0.0010.020.030.040.011 min 40 s16 min 40 s25 min 00 s33 min 20 sTime per task (faster ←)Overall (better ↑)GPT-6.2 Astra

Overall vs cost

Published cost per task attempt

Cheaper, higher scoring0.0010.020.030.040.0$1.00$2.00$5.00$10.00Cost per task (cheaper ←)Overall (better ↑)GPT-6.2 AstraGPT-6.1 SolGemini 4 ArgonGrok 4.7

Success rate (%), the mean of the 16 tasks with equal weight, 30 trials per task before XDOF's exclusions.

Tasks

16 tasks.

16 tasks · 0 with clips

Methodology

Scoring
Pass/fail + checkpoints
Time cap / attempt
30 min 00 s
Control rate
—
Models in index
10 of 10

Scoring. Annotators mark whether each trial meets the task's written success criterion, and which checkpoints it reached.

Who runs it. XDOF runs every trial on its own stations and annotates success and checkpoints.

Robots. I2RT YAM (two-arm), six stations.

Note. Coding agents only, in XDOF's harness at high reasoning effort. 30 trials per task; XDOF excludes invalidated takes, leaving 449 to 480 scored runs per model. XDOF sells evaluations and robot training data to AI labs.

Time per attempt. Published or measured for 10 of 10 models.

Cost per attempt. Published for 10 of 10 models.