Benchmarks/RoboArena

RoboArena

Real robotsPairwise preferenceReleased Jun 2025Index weight 0.81
RoboArena teamMaintainer ↗

Open-ended tasks chosen by evaluators, mostly pick and place. Ranked by pairwise preference.

As of November 17, 2026, Spirit v1.6 (j2-vla) leads with 1788, followed by DreamZero (1735) and Apricot (1661). Official leaderboard ↗ · Paper ↗

Leaderboard

Tasks
2,752
Attempts / task
—
Median cost / task
$0.023
Median time / task
25 s

Rating vs speed

Published time per task attempt

Faster, higher scoring750100012501500175015 s20 s30 sTime per task (faster ←)Rating (better ↑)DreamZeroπ0-FASTPaliGemma-binning

Cost isn’t published

RoboArena doesn’t report cost per attempt for enough models to chart. The table shows Playground’s estimates, marked est.; Methodology explains how they’re made.

Head-to-head rating from blind comparisons. Entries under 100 sessions are low-sample on the site.

#Model
1Spirit v1.6 (j2-vla)Not counted in the index178810625.0——
2DreamZero173542.6190$0.023est.25 s
3ApricotNot counted in the index166161.063.0——
4ReDROID-0.0Not counted in the index16341766.0——
5π0.5160830.7745$0.024est.26 s6π0-FAST158229.6941$0.023est.25 s7MolmoAct215791955.0——
8floppy_bagel_195k_temp1p0Not counted in the index15731528.0——
9GalaxeaVLA G0.5155969.545.0$0.018est.20 sest.
10floppy_bagel_525k_temp1p0Not counted in the index154679.033.0——
11paligemma_vq_droidNot counted in the index154629.9952——
12paligemma_diffusion_droidNot counted in the index153329.7949——
13paligemma_fast_specialist_droidNot counted in the index153129.81057——
14paligemma_fast_droidNot counted in the index151629.91062——
15damNot counted in the index149462.660.0——
16pi05_TTANot counted in the index14901479.0——
17paligemma_norm_droidNot counted in the index148190.628.0——
18floppy_bagel_420kNot counted in the index147582.729.0——
19π0146129.81120$0.023est.25 s
20floppy_bagel_285k_temp1p0Not counted in the index145812812.0——
21floppy_bagel_90k_temp1p0Not counted in the index144114510.0——
22balsomsoc (Cosmos3-Nano-Policy)Not counted in the index12993472.0——
23Phoenix12221968.0——24PaliGemma-binning78650.0629$0.019est.21 s

Tasks

2,752 evaluator-written instructions; the 30 most-run shown. Hover a task to play its clip (30 of 30).

30 tasks · 30 with clips

Graded examples

“Cover the book with the blue towel” · one A/B session

Fail · A · DreamZero: partial 0.10. Drifted toward the sofa.
Fail · B · π0.5: partial 0.55. Grabbed the white towel, then moved the blue one.

Evaluator preference: B. Neither finished; only the preference enters the ranking (Bradley-Terry with ties).

The session counts as a loss for DreamZero, which still rates higher overall (1,735 vs 1,608).

Methodology

Scoring
Pairwise preference
Time cap / attempt
27 s
Control rate
15 Hz
Models in index
6 of 24

Scoring. Evaluators run two hidden policies on a task of their choosing and pick A, B or tie.

Who runs it. Volunteer labs, blind head-to-head comparisons.

Robots. Franka (DROID setup).

Note. Entries with 20 or more sessions. The site's main board needs 100.

Time per attempt. Published or measured for 5 of 6 models. Estimated otherwise: time cap 27 s × (1 − success/2).

Cost per attempt. Not published. Estimated otherwise: one H100 at $3.29 per hour (Lambda on-demand, Oct 2026) × time. GPU price source ↗