Benchmarks/VLA-Replica

VLA-Replica

Real robotsPass/failReleased May 2026Index weight 0.72
Intelligent Robotics and Vision LabMaintainer ↗

10 tasks in pick and place, object interaction and counting, plus an out-of-distribution track.

As of November 17, 2026, π0.5 leads with 53.0, followed by GR00T N1.7 (42.0) and MolmoAct2 (39.0). Official leaderboard ↗ · Paper ↗

Leaderboard

Tasks
18
Attempts / task
15
Median cost / task
$0.069
Median time / task
76 s

Success rate (%), 15 trials per task over three rounds. Task columns are the in-distribution track.

#Model
1π0.553.039.080.087.0$0.060est.66 sest.2GR00T N1.742.035.080.087.0$0.065est.71 sest.3MolmoAct239.047.093.073.0$0.066est.72 sest.4π032.030.067.067.0$0.069est.76 sest.5SmolVLA29.027.080.020.0$0.070est.77 sest.6ACT21.08.033.07.0$0.074est.81 sest.7X-VLA13.05.027.013.0$0.077est.84 sest.
8DiT-DNot counted in the index11.08.020.013.0——
9DiT-FNot counted in the index7.04.013.00.00——

Tasks

18 tasks. Hover a task to play its clip (18 of 18).

18 tasks · 18 with clips

Graded examples

Put bread on plate · 15 trials per model

Reference scene (SO-101).
ModelSuccess± SD
MolmoAct20.930.12
π0.50.800.20
GR00T N1.70.800.00
SmolVLA0.800.20
π00.670.12
ACT0.330.12
X-VLA0.270.12

Pass when the bread rests on the correct coloured plate and the arm has returned home.

Methodology

Scoring
Pass/fail
Time cap / attempt
1 min 30 s
Control rate
30 Hz
Models in index
7 of 9

Scoring. Operator checks a one-line success criterion per trial.

Who runs it. UT Dallas IRVL runs public policies, plus submissions.

Robots. SO-101 (one arm).

Note. 15 trials per task. Every policy is fine-tuned on the same 50 demonstrations per task.

Time per attempt. Not published. Estimated otherwise: time cap 90 s × (1 − success/2).

Cost per attempt. Not published. Estimated otherwise: one H100 at $3.29 per hour (Lambda on-demand, Oct 2026) × time. GPU price source ↗