A. Definitions
A benchmark decides a matchup when Φ(gap / noise) ≥ 0.9, using the index’s published noise for each score. Each pair of policies counts once within a comparison. Intervals come from 2,000–4,000 bootstrap resamples of policies. Ratings use our refit of the index’s weighted Bradley–Terry model, which reproduces the published ratings at 0.998. We split the paper-reported results into LIBERO, SimplerEnv WidowX, SimplerEnv Google and CALVIN, and PhAIL into its two tracks, which gives 26 leaderboards from 22 benchmarks. The finishes in Figure 3 come from each benchmark’s published ranks.
Split-half test. We randomly divide the benchmark teams in two, fit the rating on each half and correlate the halves; 0.64 is the median of 200 splits. The held-out test refits the rating without each real benchmark’s team and compares its winners with single benchmarks on the same matchups.
B. Same-setup sources
Paired results come from published tables, parsed by our own scripts and checked against each paper’s reported correlations: SimplerEnv Tables IV–V (leaf tasks), PolaRiS’s per-scene data file, Genie Sim 3.0 Table I, MolmoBot Tables 6–7, RoboLab-120 v2 Table V (removed by its authors in v3 without comment), RoboDojo’s per-task leaderboard and VLA-Replica. We average Spearman correlations within tasks; the rerun reference redraws each real score from its trial count. PolaRiS is scored after its brief sim fine-tuning; the two twins without any reach 0.91, and dropping any one source leaves 0.89–0.94.
C. Each simulator against real robots
Against real-robot results from other teams. Too few shared policies to judge: RoboCasa365 (2), EBench (16), LIBERO-Plus (2) matchups. RoboTwin 2.0 has the highest agreement among simulators with enough matchups, but the differences between simulators are within noise (Appendix G). Paper-reported LIBERO and SimplerEnv Google show no positive link to other teams’ robots, although SimplerEnv ranks policies well against the setups it mirrors.
D. Model-family forecast
Families follow the index’s model type. For each real benchmark, the forecast is the index rating refitted on eight team-run simulators (RoboDojo sim, RoboTwin 2.0, RoboLab-120, MolmoSpaces, RoboCasa365, RoboColiseum, EBench and Embodied First Exam), leaving out every benchmark from the target’s team. Rating gaps convert to win probabilities with the Elo curve; observed win probabilities are Φ(gap / noise). The real-robot excess of +22 points holds when we drop any one WAM (+18 to +25) or benchmark (+18 to +34), weaken the rating prior (+15), add paper-reported suites (+25), restrict to VLAs with equally thin simulation coverage (+24), or restrict to VLAs released after the first WAM (+34). LLM coding agents beat the forecast against VLAs by similar margins in simulation and on real robots (+15 and +16 points), which places their error in the forecast rather than the transfer. The like-for-like comparison in Section 6 uses the four WAMs with forecast pairs in both domains (Cosmos3-Nano-Policy, DreamZero, InternW0-Δ, OpenWAM-α). Developers choose which checkpoints reach some real benchmarks. On RoboDojo, where the benchmark team chooses and trains every policy, WAMs win all 21 matchups against VLAs, against a forecast of 88%; on benchmarks that run developer-supplied checkpoints they win all 17, against 64%. Removing every LLM agent from every benchmark and refitting leaves the real-robot excess at +22 points. Four policies on RoboDojo (DM0, InternVLA-A1, SLA-Preview, Spirit v1.5) have no outside simulation results and drop out of its forecast.
Share of real-robot matchups a real benchmark decides in which the simulator’s score gap picks the real winner.
E. Design comparisons
Versions of the same benchmark on the same policies and real matchups, changing one design choice at a time. These comparisons are observational, not randomized.
F. Robustness
Without a confidence threshold, so that close calls count, sim ↔ real agreement is 68% over 442 matchups and real ↔ real 60% over 120 (difference +9 points, 90% CI −12 to +31). Leaving out one benchmark at a time keeps sim ↔ real agreement between 67% and 78%.
LLM agents. Excluding every matchup that involves an LLM agent leaves sim ↔ real agreement at 73% (224 matchups) and sim ↔ sim at 93%. Real ↔ real agreement falls from 70% to 61% over 55 matchups (90% CI 25–91%), within its wide interval. Refitting the index without the three benchmarks that test only agents (AA-Robotics, DataCenterBench and the simulated Embodied First Exam) leaves four agents in the top ten, as in the full refit. Trial shares exclude the two pairwise-rated benchmarks. Agent entries are counted per benchmark; the figure counts RoboColosseum’s two tracks and PhAIL’s two task sets separately.
G. Ranking engines and single benchmarks
The data cannot rank physics engines. Pooled by engine, simulators pick another team’s real winner 74% of the time on Isaac Sim and Isaac Lab (4 benchmarks, 78 matchups), 76% on SAPIEN (3, 34) and 66% on MuJoCo (5, 29). A permutation test finds no difference (p = .58). The spread within an engine exceeds the spread between engines: RoboTwin 2.0 and paper-reported SimplerEnv Google both run on SAPIEN and agree 95% and 43% of the time. Who runs a benchmark and how it is built matter more than its physics engine. Engines follow each benchmark’s documentation; MolmoSpaces assets also load in other simulators, and its benchmark runs in MuJoCo.
Nor can the data rank single simulators by how well they predict real robots. Each simulator decides 2 to 29 matchups against other teams’ robots. Telling 95% agreement from 70% with 80% power takes about 71 matchups per benchmark once shared policies are accounted for, and telling 85% from 70% takes about 259. Differences across the 14 simulators fall short of significance (p = .08) under a permutation test that, ignoring shared policies, already overstates them. Direct comparison fails for the same reason: only 3 of 58 simulator pairs disagree with each other on five or more of the matchups they share.
A weaker question has an answer: which benchmarks agree with the rest of the index. We refit the index without each benchmark’s team and score the benchmark’s decided matchups against the refit, using every matchup it decides rather than only those a real robot also decides. Expected agreement comes from the refit’s rating gap, so a benchmark that compares only distant policies earns no credit for it. RoboTwin 2.0 agrees 14 points more often than expected over 164 matchups; RoboChallenge agrees 48 points less often over 25. MolmoSpaces and RoboCasa365 also clear expectation on fewer matchups. Embodied First Exam and DataCenterBench clear it on 12 matchups each between coding agents rated far apart, which we do not count as evidence. Every other benchmark, real or simulated, is indistinguishable from the rest.
Agreement with an index refitted without the benchmark’s team, on matchups the benchmark decides and the refit rates. Expected agreement is a logistic function of the refit’s rating gap, fitted on all benchmarks together; intervals resample policies. Agreement with the index is not agreement with real robots. The rest of the index is mostly simulation, and a benchmark that shares the simulators’ blind spots scores well here. RoboChallenge differs from the index in design as well as outcome.
H. Checks added after review
Coverage, same policies. For the 10 policies with two or more simulation and two or more real benchmarks, we refit the simulation-only and real-only ratings 200 times, each time keeping one randomly drawn benchmark of each kind for these policies (every other policy keeps all its results). The correlation falls from 0.87 to a median of 0.45, and 99% of draws fall below the full-coverage value.
Benchmark-pair features. Across all 701 decided cross-team matchups, simulated and real, agreement does not rise when the two benchmarks share a robot, both train every policy with one recipe, or score the same way. In a joint logistic fit only the pair’s kind matters: simulator–real and real–real pairs agree less than simulator–simulator pairs. The features are coarse, and the analysis shows association only.
WAM success rates. On benchmarks scored in percent, for the 30 WAM–VLA pairs also observed in simulation, WAMs score 32 points higher on real robots (90% CI 28–38) and 21 points higher in simulation (14–33). The real-robot figure rests on three benchmarks.
Close calls in forecasts. Hiding each policy’s real results and forecasting it from simulation alone, the index picks the real winner in 57% of the 141 matchups where the two policies are rated within 150 points of each other.
I. Limitations
- Real-to-real agreement rests on 60 matchups (90% CI 35–100%). The matched comparison on shared matchups and the RoboDojo test carry the setup result alongside it.
- The coverage comparison in Figure 4 rests on 10 well-covered policies, and its intervals overlap (0.64–0.99 against 0.36–0.77).
- Counting only decided matchups favors large gaps. Twins never confidently pick the wrong winner; close calls between strong policies still need many real trials, and simulation-based forecasts for closely rated policies are little better than a coin flip (Appendix H).
- Each twin was validated by its own builders on three to six policies per task, mostly older policies in SimplerEnv and checkpoints of a single model in Genie Sim 3.0. None of the paired datasets includes a WAM; for WAMs, same-setup evidence is RoboDojo alone, whose simulator calls all 20 matchups between its two WAMs and VLAs correctly. An independent rerun of one mirrored twin, on policies its builders did not test, is the most direct next test.
- The model-family result rests on 7 WAMs and 38 real matchups; its size is uncertain (+12 to +35 points). Developers choose where to submit on some real benchmarks. RoboDojo trains every policy itself, and WAMs lead there too.
- Several DROID benchmarks share policies and reference data, which makes their comparisons less independent than their count suggests. Coding agents’ results mix harness and model. Agreement between benchmarks is not ground truth.