Benchmarks/Embodied First Exam

Embodied First Exam

SimulationCode checkReleased Oct 2026Index weight 0.95
Embodied First Exam teamMaintainer ↗

Standard, real-time and privileged modes. Coding agents only.

As of November 17, 2026, Claude Fable 5.2 leads with 49.6, followed by GPT-6 Astra (34.4) and GPT-6.1 Sol (32.2). Official leaderboard ↗ · Paper ↗

Leaderboard

Tasks
498
Attempts / task
1
Median cost / task
$5.29
Median time / task
1 h 12 min

Standard vs speed

Published time per task attempt

Faster, higher scoring0.0020.040.050 min 00 s1 h 23 min1 h 57 minTime per task (faster ←)Standard (better ↑)Claude Fable 5.2Claude Opus 5.5

Standard vs cost

Published cost per task attempt

Cheaper, higher scoring0.0020.040.0$0.500$1.00$2.00$5.00$10.00$20.00Cost per task (cheaper ←)Standard (better ↑)Claude Fable 5.2Claude Opus 5.5Gemini 4 ArgonQwen3.8 MaxDeepSeek V4.1 Flash

Share of tests solved (%). Privileged mode gives the agent extra state information.

Tasks

498 tasks. Hover a task to play its clip (498 of 498).

498 tasks · 498 with clips

Graded examples

Turn off the sink faucet · standard mode

Pass · Claude Opus 5.5: pass. 5,174 control steps, 1,737 s.
Fail · GPT-6 Astra: fail. Ran to the 3,600 s cutoff.
AgentStandard
Claude Opus 5.5Pass
Gemini 4 ArgonPass
Kimi K3.1 ProPass
DeepSeek V4.1 FlashPass
GPT-6 AstraFail
GLM 5.3 FlashFail
Qwen3.8 MaxFail
Gemini 3.8 FlashFail

Check: the faucet handle ends outside 0.40–π rad after the last step, and the task was not already solved at the start.

Methodology

Scoring
Code check
Time cap / attempt
1 h 00 min
Control rate
—
Models in index
10 of 10

Scoring. Simulator success function after the agent's last step, pass/fail.

Who runs it. Benchmark team runs every entry.

Robots. 44 simulated robots across 31 suites.

Note. Only coding agents are tested here.

Time per attempt. Published or measured for 9 of 10 models. Estimated otherwise: time cap 3600 s × (1 − success/2).

Cost per attempt. Published for 9 of 10 models. Estimated otherwise: GPT-6.1 Sol's median published cost per attempt on other boards.