Benchmarks/Paper-reported

Paper-reported

SimulationCode checkReleased 2021 to 2024Index weight 0.28
Ai2Maintainer ↗

LIBERO's four suites (spatial, object, goal, long), SimplerEnv robots, CALVIN multi-step chains.

As of November 17, 2026, X-VLA leads with 98.1, followed by OpenVLA-OFT (97.1) and GR00T N1.7 (97.0). Official leaderboard ↗

Leaderboard

Tasks
105
Attempts / task
—
Median cost / task
$0.0086
Median time / task
9.5 s

Median of papers' reports, or the model's own paper. Only models matched on other boards are shown.

Tasks

105 tasks. Hover a task to play its clip (105 of 105).

105 tasks · 105 with clips

Methodology

Scoring
Code check
Time cap / attempt
18 s
Control rate
—
Models in index
13 of 13

Scoring. Each paper's standard simulator checks; CALVIN counts completed subtasks.

Who runs it. Self-reported in papers, collected by AllenAI.

Robots. Simulated Franka, Google Robot, WidowX.

Note. Only models matched elsewhere are used. Lowest weight in the ranking.

Time per attempt. Not published. Estimated otherwise: time cap 18 s × (1 − success/2).

Cost per attempt. Not published. Estimated otherwise: one H100 at $3.29 per hour (Lambda on-demand, Oct 2026) × time. GPU price source ↗