Characterizing the
Sim-to-Real Gap

We measure the sim-to-real gap across the 22 public benchmarks behind the Robot Intelligence Index: 2,162 published scores for 85 policies, and 274 paired simulated and real results from seven validation studies. We find that rankings from simulators matched to a real setup agree closely with the real-robot ranking, and that substantial ranking variation occurs across benchmark setups (environment, embodiment, assets) in both simulated and real-world evals.

MeasureResultCompared with
Same setup
Mirrored simulator vs. its real setupRank correlation, builder-run studies0.910.92Modeled repeat of the real test
Across setups
Simulator vs. another team’s real robotSame winner, 260 matchups73%90% CI 59–86%70%Two real benchmarks from different teams, 60 matchups, 90% CI 35–100%
Aggregation
Two halves of the indexRank correlation0.640.50Two single benchmarks
Simulation vs. real-robot rating10 policies on two or more benchmarks of each kind0.890.6127 policies on one benchmark of a kind
The same 10 policies, one benchmark of each kindOur refit, median of 200 draws0.450.87Full coverage, our refit
New model families
World-action models (WAMs) vs. VLAsReal-robot matchups won38 of 3878%Simulation forecast

In mirrored studies, we find that simulated and real rankings agree as closely as a repeat of the real-robot evaluation would. When setups vary, agreement falls for simulation and real robots alike: simulation evals pick the same winner as a separate team’s real-robot evals at approximately the same rate as two real-robot benchmarks from separate teams.

Aggregating benchmarks stabilizes rankings. Rankings built from two disjoint halves of our index correlate at 0.64, against 0.50 for two single benchmarks from different teams. Holding the policies fixed shows the effect of coverage: for the ten best-covered policies, simulation-only and real-only ratings from our refit of the index correlate at 0.87, and rebuilding each rating from one benchmark of each kind lowers that to 0.45.

Real robot evals remain key for benchmarking success rates, identifying shared errors in simulation, and for evaluating new model families. WAMs, for example, win every one of their real-robot matchups against VLAs, far more often than simulation forecasts. Real-robot testing is also unevenly spread: LLM agents make up 27% of real-robot entries against 12% of simulation entries, mostly through two small benchmarks that test only agents, and no benchmark running 300 or more trials per model tests one.

1. Introduction

Separating two policies 10 points apart takes hundreds of real trials each (STEP 2025; Kress-Gazit et al. 2024), and nearly every new policy reports simulation scores instead. Whether those scores rank policies the way real robots would is the evaluation form of the reality gap (Jakobi et al. 1995; Koos et al. 2013).

Prior work tests one simulator at a time. Each study validates a simulator against its builders’ own robot, usually on four to nine policies, and reports correlations near 0.9 (Kadian et al. 2020; Li et al. 2024; PolaRiS 2025; Genie Sim 3.0 2026). Two questions remain open. How well do simulators predict real robots run by other teams, on the policies the field actually uses? And how reliable is the real reference itself?

We answer both with the data behind the Robot Intelligence Index, which collects every public robot leaderboard active in 2026. We compare benchmarks on the matchups both decide confidently, and group each comparison by whether the setups match (a simulator that mirrors a real setup, or one team’s simulator and robots running the same tasks and policies) and by whether each side is simulated or real. Figure 1 states our results. When the setup matches, simulation predicts real-robot results closely. When it does not, simulators and other teams’ real-robot benchmarks miss at similar rates, picking the wrong winner in roughly a quarter to a third of matchups.

Figure 1. How often a result predicts the winner on a real robot
Simulation predicting realReference
Same setup
Mirrored simulator→ its real twin · 107 matchups
100%≥97% at 95%
RoboDojo simulator→ a held-out RoboDojo arm · 107 matchups
94%80–100%
RoboDojo real arm→ another RoboDojo arm · 172 matchups
97%87–100%
Different teams, different setups
Public simulator→ another team’s real robot · 260 matchups
73%59–86%
Real robot→ another team’s real robot · 60 matchups
70%35–100%
Simulator→ another team’s simulator · 381 matchups
93%shared errors, §6
Same winner, share of decided matchups
playground/Robot Intelligence Index

Share of matchups, decided by both sources, in which the predictor picks the real benchmark’s winner. 90% intervals from resampling policies.

The result changes what a leaderboard should measure. If a policy’s rank depends on its setup, no single benchmark, simulated or real, measures how good the policy is. Our index aggregates comparisons across setups for this reason. The sections below show that simulation ranks correctly when the setup matches (Section 3), that setups differ enough to reorder policies (Section 4), that aggregation recovers a stable ranking (Section 5), where real robots remain necessary (Section 6), and how real-robot testing is spread across model families (Section 7).

2. Data and method

Our data is the index snapshot of November 9, 2026 (Table 1), plus every published study that scores the same policies on the same tasks in simulation and on a real robot.

DataCount
Benchmarksscored as 14 simulation and 12 real-robot leaderboards22
Published scoresoverall, task and category scores in 394 leaderboard rows2,162
Policies280 policy–benchmark results, matched by name and version85
Matchups within benchmarks1,510 decided with 90% confidence1,862
Matchups two teams both decide381 sim–sim, 260 sim–real, 60 real–real701
Real-robot trials behind the scoresexcluding RoboArena’s pairwise A/B evaluations15,516
Paired simulated and real resultssame policy and task, from seven published studies274
Cross-family matchups38 WAM vs. VLA on real robots374
Sources audited for same-setup datapapers, appendices and code27

Table 1. Robot Intelligence Index snapshot, November 9, 2026, and the published validation studies.

Our unit of comparison is the matchup: two policies scored on the same benchmark. A benchmark decides a matchup when its score gap would survive a rerun at least 90% of the time, under the index’s noise model. Two benchmarks agree on a matchup when both decide it and pick the same winner. We never compare two benchmarks run by the same team, except inside RoboDojo, where we hold the team fixed on purpose. For paired results we report the rank correlation of simulated and real scores within each task. The appendix gives details and robustness checks.

3. Matched simulators closely track real rankings

We audited all 27 sources in the index, including papers, appendices and code, for the same policies scored in simulation and on a matching real setup. Three simulators mirror a specific real setup: SimplerEnv, PolaRiS and Genie Sim 3.0. Together they provide 98 paired results over 21 tasks and 16 policies.

Figure 2. Mirrored simulators track real rankings
Mirrored simulatorComparison
Simulator mirrors the real scene
SimplerEnvGoogle Robot, visual matching
0.880.67–0.93
PolaRiSsplat replicas of real scenes
0.910.81–0.93
Genie Sim 3.0Agibot G1, mirrored scenes
0.990.97–1.00
All mirrored twins98 paired results
0.910.81–0.94
Simulator does not mirror the scene
Unmatched simulatorsame real tasks
0.750.48–0.85
Shared task names onlypolicies trained separately
0.490.30–0.66
Real against real
Modeled rerun of the real testsimulated from trial counts
0.92expected
Rebuilt real labVLA-Replica, 5 trials per cell
0.650.53–0.88
Within-task rank correlation with the real result
playground/Robot Intelligence Index

Mean Spearman correlation of simulated and real scores within each task, with 90% intervals.

In the mirrored studies, mean within-task rank correlation is 0.91, close to the 0.92 expected for a repeat evaluation under our trial-count noise model. All 107 matchups confidently decided by both simulation and reality agree. This supports simulation as a relative evaluation tool in these matched settings, especially for clearly separated policies. It does not establish equivalent reliability for close comparisons or unseen model families: the rerun reference is modeled, and the mirrored studies were conducted by their builders on limited policy sets.

Two measured references complement the modeled one. A second real arm inside RoboDojo agrees with a held-out arm in 97% of decided matchups, while the one study that rebuilt a real lab elsewhere, with four policies and five trials per cell, reproduced the original ranking at only 0.65 (VLA-Replica 2026). How closely the simulator matches the real setup is the factor these studies vary directly. On the same real tasks, unmatched simulators reach 0.75, and simulators that share only task names reach 0.49.

Simulation also tracks real results closely inside one team’s protocol. RoboDojo trains 12 policies with one recipe and evaluates them in simulation and on three real arms. Its simulator predicts a held-out arm in 94% of decided matchups, against 97% for another real arm.

Within these studies, what the policy sees appears to drive the match. PolaRiS’s photoreal replicas correlate with real results at only 0.30 before a brief fine-tuning step in simulation, and its builders trace the error to small visual differences. SimplerEnv rises from 0.67 to 0.88 when its visuals match the real scene.

4. Rankings vary across benchmarks

Most simulators mirror no real setup, and most real benchmarks share none with each other. Across teams, agreement falls for both kinds of evidence (Figure 1). A public simulator agrees with another team’s real robot on 73% of 260 decided matchups. Two real-robot benchmarks from different teams agree on 70% of 60. On the matchups where both are available, a simulator and another team’s real benchmark predict a target real benchmark at similar rates (72% and 73%). Similar is not the same as equivalent. The difference is −2 points with a 90% interval of −25 to +12, which cannot exclude simulation being up to 25 points worse. Pinning the difference to within 10 points would take roughly 160 shared matchups, against 46 today. The only earlier pairwise figure, from Habitat navigation, sits close to ours at about 75% (Kadian et al. 2020).

The disagreements are not noise. Score noise accounts for 2 points of the sim–real disagreement, and in 66% of the matchups where simulation and reality pick opposite winners, the simulator was at least 99% sure. We isolate the noise share by computing, for each of the 260 sim–real matchups, the probability that a rerun of either benchmark would reverse its call under the index’s noise model, and averaging. They reflect systematic differences between benchmarks. Coarse features of a benchmark pair do not account for them: pairs that share a robot, a training recipe or a scoring type agree no more often than pairs that do not (Appendix H).

Figure 3. One policy, many rankings
SimulationReal robot
GPT-6 Astra8 benchmarks
38point swing
GPT-6.1 Sol4 benchmarks
21point swing
Gemini 4 Argon5 benchmarks
39point swing
OpenWAM-α4 benchmarks
40point swing
Claude Opus 5.56 benchmarks
61point swing
π0.517 benchmarks
100point swing
GalaxeaVLA G0.55 benchmarks
46point swing
MolmoAct29 benchmarks
78point swing
Xiaomi-Robotics-04 benchmarks
35point swing
GR00T N1.711 benchmarks
88point swing
Fast-WAM4 benchmarks
65point swing
X-VLA8 benchmarks
89point swing
π0-FAST6 benchmarks
66point swing
OpenVLA-OFT5 benchmarks
90point swing
π015 benchmarks
83point swing
Diffusion Policy4 benchmarks
50point swing
SmolVLA4 benchmarks
50point swing
ACT4 benchmarks
34point swing
Finish on each benchmark, from last to first
playground/Robot Intelligence Index

Each policy’s finish on each benchmark, from last to first. Policies with four or more benchmarks, ordered by index rating.

The effect on rankings is large. Among the 32 policies tested on three or more benchmarks, the median gap between best and worst finish is 43 percentile points. Of the 22 policies that place top three somewhere, 11 place in the bottom half elsewhere. π0.5 finishes first on ArmnetBench and last on PhAIL.

Cross-setup disagreement is substantial in both simulated and real evaluation. A result from another real lab is therefore not automatically a more reliable guide to a target benchmark than a simulation result. These comparisons do not isolate the contribution of physics fidelity: benchmarks also differ in tasks, embodiments, policy training, and evaluation protocols. Our evidence supports treating benchmark context as a major source of ranking variation, without identifying which differences cause it.

5. Aggregating benchmarks stabilizes rankings

If performance depends on setup, the quantity worth measuring is performance across setups. This is what motivates the Robot Intelligence Index. We view the index as a first step towards aggregate performance measurement. It compares policies only within each benchmark, where the setup is shared, and fits one rating per policy to all comparisons across 22 benchmarks with a weighted Bradley–Terry model (Bradley and Terry 1952). Two tests compare the aggregate with single benchmarks (Figure 4), and a third holds the policies fixed.

Figure 4. Aggregated rankings agree more than single benchmarks
IndexComparison
Two independent rankings of the same policies
Two single benchmarksdifferent teams · 143 pairs
0.50median
Two halves of the indexdisjoint teams · 200 splits
0.640.48–0.76
Index simulation rating against real-robot rating
One benchmark of a kind27 policies
0.610.36–0.77
Two or more of each10 policies
0.890.64–0.99
Rank correlation between the two rankings
playground/Robot Intelligence Index

Spearman rank correlations. Top: two rankings built from separate benchmarks. Bottom: the index fitted on simulation only and on real robots only, for the 37 policies in its default ranking.

Ratings built from disjoint halves of the benchmark teams have a median correlation of 0.64, compared with 0.50 between individual benchmarks from different teams. Simulation-only and real-only ratings correlate at 0.89 among ten policies evaluated on at least two benchmarks of each kind, versus 0.61 among 27 policies with thinner coverage. These patterns are consistent with aggregation improving ranking stability, as in machine-learning benchmarking, where an aggregate reference ranks models more stably than any single benchmark (Perlitz et al. 2024). However, the coverage groups contain different policies and their uncertainty intervals overlap, so that comparison alone does not show that coverage produces the improvement. Holding the policies fixed does. For the same ten well-covered policies, rebuilding each rating from one randomly drawn benchmark of each kind lowers the correlation from 0.87 in our refit to a median of 0.45 (90% of draws between 0.09 and 0.78). The results support aggregation as a promising measurement strategy whose reliability should be tracked as coverage expands.

6. Where real robots remain necessary

Aggregation averages over differences between benchmarks. Four things still need real-robot results.

Success rates. Mirrored simulators track the order more closely than the level. On average they underestimate real success rates by 11 points (90% CI 9–16), as earlier manipulation twins do (Li et al. 2024). This average measures directional bias, not absolute error: larger errors on individual tasks can offset one another. Deployment decisions need real trials.

Errors simulators share. Simulators from different teams agree with each other on 93% of decided matchups, far more often than they agree with real robots, and part of that agreement is shared error. On the 56 real-robot matchups two or more simulators decide, a simulator picks the wrong winner 19% of the time, and 64% of the time when another simulator is already wrong. Eleven of the 15 shared mistakes over-rate X-VLA or MolmoAct2, the leaders of the saturated LIBERO, SimplerEnv and CALVIN suites (LIBERO-PRO 2025; LIBERO-Plus 2025). Adding simulators does not average this error out.

New model families. Our simulation-based rating forecast underestimates WAMs’ pairwise wins against VLAs on the available real benchmarks (Figure 5). WAMs win all 38 observed matchups, compared with a forecast of 78%. On other teams’ simulators, the WAMs tested on real robots also beat their forecast, by 13 points, while WAMs never tested on real robots match theirs (+2). Compared like for like, the discrepancy is 10 points larger on real robots (90% CI −4 to +31). The forecast understates these WAMs everywhere, and most on real robots. The WAM gap persists when we compare only with VLAs that have equally thin simulation coverage, compare only with VLAs released after the first WAM, or drop any one WAM or benchmark (Appendix D). LLM agents beat their forecast against VLAs by similar margins in simulation and on real robots (+15 and +16 points), so for them the error lies in the forecast, not the transfer. In success rates, WAMs lead the same VLAs by 32 points on real robots and 21 points in simulation (Appendix H). This is evidence of a family-associated forecast error in our sample. These checks narrow the competing explanations, but they do not establish whether the cause is simulation fidelity, coverage, or differences in the evaluated tasks and checkpoints.

Figure 5. Simulation-based forecasts understate WAMs
Simulation forecastObserved in simObserved on real robots
WAMs against VLAs
In simulation244 matchups
+568 → 73%
On real robots38 matchups, 38 won
+2278 → 99%
Each WAM against VLAs on real robots
Dream-1RoboDojo-RC, 2 of 2 won
+5347 → 100%
DreamZeroRoboArena, 7 of 7 won
+4057 → 96%
Cosmos3-Nano-PolicyPhAIL, 4 of 4 won
+3070 → 100%
FLUX 3 ActionPhAIL, 4 of 4 won
+1981 → 100%
OpenWAM-αRoboDojo, 7 of 7 won
+1387 → 100%
InternW0-ΔRoboDojo, 7 of 7 won
+1288 → 100%
MWA-PhoenixRoboDojo, 7 of 7 won
+1090 → 100%
Win rate: predicted from other teams’ simulators (ring) and observed (dot)
playground/Robot Intelligence Index

WAMs’ win rate against VLAs, forecast by the index’s rating fitted on other teams’ simulators. Values are observed minus forecast, in points.

For WAMs, evals in sim often identify the winner, while the aggregate forecast understates WAMs’ frequency of pairwise wins. Individually they pick the real winner in 27 of 29 of these matchups, against 57 of 69 VLA-against-VLA matchups on the same simulators. The miss is largest where a model has one outside simulator: Dream-1, rated only by RoboColiseum, was forecast at 47% and won every real matchup. A matched simulator does better. RoboDojo’s own simulator ranks its two WAMs first and second, as its real arms do, and decides all 20 of their matchups against VLAs correctly. Real results also fix the index: with them, WAMs outrate 89% of co-rated policies, as on real robots (89%); simulation alone puts them at 82%.

A specific robot. No aggregate predicts the quirks of one lab’s robot. On a held-out real benchmark, the index and a single other benchmark pick the winner equally often (68% and 71%). To know how a policy performs on a particular robot, tests with that specific setup are needed. RoboColosseum takes a first step in this direction by rating policies separately on each robot it runs: a Franka, a two-arm YAM and an SO-101.

Matched to a real setup, simulation tracks relative rankings closely. Real robot results set success rates, catch the errors simulators share, and test model families simulators were not built for.

7. LLM agents are over-represented on small real-robot benchmarks

Real-robot tests are scarce, so which policies receive them shapes what real results can show. The index covers 18 LLM agents: general-purpose language models, such as GPT-6 Astra and Claude Opus 5.5, that control a robot through a software harness. By count, agents receive more than their share of real-robot testing. They are 21% of the policies with a result and 12% of simulation entries, but 27% of real-robot entries. Of the 18 agents, 13 have a real-robot result, against 23 of 42 VLAs and 7 of 20 WAMs.

The surplus reflects which benchmarks test agents, not how they are tested: each benchmark runs all its policies for the same number of trials. Agents appear on 4 of the 11 real-robot benchmarks (Figure 6). Two of them, AA-Robotics and DataCenterBench, test only agents, at 8 and 30 trials per model, the fewest of any real-robot benchmark. Without these two, agents are 14% of real-robot entries, below their share of policies. Two evaluators, Robocurve and Artificial Analysis, supply 21 of agents’ 24 real-robot entries, and no benchmark running 300 or more trials per model tests an agent.

Figure 6. LLM agents are tested mostly on the smallest real-robot benchmarks
LLM agentRobot policy (VLA, WAM or baseline)
Fixed number of trials per model
AA-Robotics8 trials per model
5 of 5agents
DataCenterBench30 trials per model
9 of 9agents
PhAIL (pick and place)100 trials per model
0 of 4agents
RoboDojo-RC120 trials per model
7 of 10agents
VLA-Replica150 trials per model
0 of 7agents
RoboDojo (real)180 trials per model
0 of 14agents
PhAIL (DROID tasks)286 trials per model
0 of 6agents
RoboChallenge300 trials per model
0 of 11agents
ArmnetBench360 trials per model
0 of 7agents
RoboColosseum (fine-tune)500 trials per model
0 of 5agents
Pairwise ratings
RoboColosseum (open)pairwise ratings
3 of 8agents
RoboArenapairwise ratings
0 of 8agents
playground/Robot Intelligence Index

Each dot is one policy on one real-robot benchmark. Benchmarks are ordered by real trials per model. RoboArena and RoboColosseum’s open track rate policies by pairwise comparison instead.

The concentration thins the evidence behind agents’ real-robot results in two ways. Their entries carry 7% of the real trials behind the index, so each result is less certain, and the index’s noise model weights it less. And the comparison that places agents against robot policies on the same robot rests on two benchmarks, RoboDojo-RC and RoboColosseum’s open track, with 36 head-to-head pairs between them. Our findings do not depend on agents: excluding every matchup that involves one, simulation agrees with another team’s real robot on 73% of decided matchups, the same rate as with them. Nor does agents’ standing depend on the agent-only benchmarks: refitting the index without them keeps four agents in its top ten (Appendix F). Real-robot coverage is thinnest for the family real-robot forecasts miss most: only 7 of 20 WAMs have a real-robot result (Section 6).

LLM agents are over-represented on real robots by count, not by evidence. Most of their real-robot results come from two small benchmarks that test only agents, and the large real-robot benchmarks test none. Until those benchmarks include agents, how LLM agents compare with robot policies on real hardware rests on two benchmarks.

8. Implications for our index methodology

The index already compares policies only within benchmarks and aggregates across benchmarks, the design these results favor. Based on this analysis, we’ll be rolling out four changes to our index methodology in the coming weeks:

  • Treat simulators as related evidence. Ten simulators that share a blind spot are not ten independent votes. The overall rating will weight real-robot results more, and rank ranges will resample related simulators together.
  • Weight each simulator by how well it tracks held-out real results (Appendix C) once shared real matchups can tell simulators apart (Appendix G), alongside evaluator independence and repeatability.
  • List per-task checkpoints separately, such as the models some developers fine-tuned for each RoboChallenge task.
  • Publish the index’s sim-to-real agreement with every update, by model family, starting with the numbers here.

For benchmark builders our evidence suggests that evaluation requires measurement of the same models across many different environments. Our results also imply that having more simulation episodes does not help resolve the sim2real disagreement, given that most of the sim2real misses were already high-confidence (Appendix E).

9. Conclusion

In the studies we analyzed, matched simulators often reproduce real-policy rankings closely. Across teams and setups, both simulated and real benchmarks disagree, making a single benchmark an incomplete measure of policy performance. Aggregating benchmarks yields encouraging ranking consistency, although coverage and shared errors limit what that agreement establishes. Real-robot evaluation remains necessary to validate absolute success rates and detect failures of simulation-based forecasts, particularly for new model families. The practical goal is to measure when simulation predicts real outcomes reliably and where additional real evidence changes the assessment.

References

  1. Bradley, R. A., and Terry, M. E. (1952). Rank analysis of incomplete block designs. Biometrika 39.
  2. Genie Sim 3.0 (2026). Simulation benchmark behind AgiBot’s RoboColiseum. arXiv:2601.02078.
  3. Jakobi, N., Husbands, P., and Harvey, I. (1995). Noise and the reality gap: The use of simulation in evolutionary robotics. European Conference on Artificial Life.
  4. Kadian, A., et al. (2020). Sim2Real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters. arXiv:1912.06321.
  5. Koos, S., Mouret, J.-B., and Doncieux, S. (2013). The transferability approach: Crossing the reality gap in evolutionary robotics. IEEE Transactions on Evolutionary Computation. arXiv:1307.1870.
  6. Kress-Gazit, H., et al. (2024). Robot learning as an empirical science: Best practices for policy evaluation. arXiv:2409.09491.
  7. Li, X., et al. (2024). Evaluating real-world robot manipulation policies in simulation (SimplerEnv). arXiv:2405.05941.
  8. LIBERO-PRO (2025). Perturbation tests of the LIBERO suite. arXiv:2510.03827.
  9. LIBERO-Plus (2025). Camera, layout and initial-state perturbations of LIBERO. arXiv:2510.13626.
  10. Perlitz, Y., et al. (2024). BenchBench: benchmark agreement testing for language models. arXiv:2407.13696.
  11. Playground (2026). Introducing the Robot Intelligence Index. withplayground.ai.
  12. PolaRiS (2025). Real-to-sim evaluation with Gaussian-splat replicas of real DROID scenes. arXiv:2512.16881.
  13. STEP (2025). Sequential testing for robot policy evaluation. arXiv:2503.10966.
  14. VLA-Replica (2026). A real-robot benchmark rebuilt in a second lab. arXiv:2605.20774.

Appendix: method, robustness and limitations

A. Definitions

A benchmark decides a matchup when Φ(gap / noise) ≥ 0.9, using the index’s published noise for each score. Each pair of policies counts once within a comparison. Intervals come from 2,000–4,000 bootstrap resamples of policies. Ratings use our refit of the index’s weighted Bradley–Terry model, which reproduces the published ratings at 0.998. We split the paper-reported results into LIBERO, SimplerEnv WidowX, SimplerEnv Google and CALVIN, and PhAIL into its two tracks, which gives 26 leaderboards from 22 benchmarks. The finishes in Figure 3 come from each benchmark’s published ranks.

Split-half test. We randomly divide the benchmark teams in two, fit the rating on each half and correlate the halves; 0.64 is the median of 200 splits. The held-out test refits the rating without each real benchmark’s team and compares its winners with single benchmarks on the same matchups.

B. Same-setup sources

Paired results come from published tables, parsed by our own scripts and checked against each paper’s reported correlations: SimplerEnv Tables IV–V (leaf tasks), PolaRiS’s per-scene data file, Genie Sim 3.0 Table I, MolmoBot Tables 6–7, RoboLab-120 v2 Table V (removed by its authors in v3 without comment), RoboDojo’s per-task leaderboard and VLA-Replica. We average Spearman correlations within tasks; the rerun reference redraws each real score from its trial count. PolaRiS is scored after its brief sim fine-tuning; the two twins without any reach 0.91, and dropping any one source leaves 0.89–0.94.

Sourcesimulation · realSame setup?What existsFine-tuned in sim?Used
Ranked, simulation and real
RoboDojoARX X5 · ARX X5, Piper, Piper XSame task namesSame team and policies, trained separately on sim and real demos. 7 task names in both tracks, not aligned by design; 2 on the same robot.arXiv 2607.04434 Table 2; robodojo-benchmark.comSeparate weights: sim demos for sim, real demos for real70 cells
Ranked, simulation
RoboTwin 2.0ALOHA-AgileX · sim-trained policies deployedNoReal experiments test policy transfer; the real-tested checkpoints have no sim scores.arXiv 2506.18088 Table 4––
RoboLab-120Franka (DROID) · 4 policies, v1 and v2 onlySame task namesMatched sim and real success on six unnamed simple tasks for four policies, as one aggregate. In v1 and v2; removed from v3 without comment and replaced by a comparison with RoboArena. Cited as v2, which never changes.arXiv 2604.09860v2 Table VNone (DROID-trained policies)4 results
MolmoSpacesFranka FR3 · Franka (DROID), MolmoBot paperSame task namesSame checkpoints in sim and real on related pick-and-place tasks; scenes not mirrored.arXiv 2603.16861v2 Tables 6, 7None: sim-trained MolmoBot runs zero-shot on the real robot12 cells
RoboCasa365mobile Franka · sim-trained policies deployedNoPolicy transfer only.arXiv 2603.04356 Table 5––
RoboColiseum (Genie Sim 3.0)Agibot G1 and G2 · Agibot G1 (platform paper only)Mirrored twinIts platform paper scores four pi0.5 checkpoints per task in mirrored sim and real scenes on an Agibot G1. RoboColiseum's own leaderboard runs on the G2 and has no real counterpart.arXiv 2601.02078v4 Table ISome checkpoints trained on sim data, some on real16 cells
EBenchIsaac Sim arms · –NoSim only; sim-real correlation left to future work.arXiv 2606.18239––
LIBERO-Pro2.0Franka · –NoSim only.arXiv 2610.24587––
LIBERO-PlusFranka · –NoSim only.arXiv 2510.13626––
Embodied First Exam44 robots · –NoSim only; its sim-to-real suite deploys to a second simulator.embodied-first-exam.ai––
Ranked, paper-reported
SimplerEnvGoogle Robot, WidowX · Google Robot, WidowXMirrored twinSim scenes built to copy the real setups; same checkpoints in both. Visual matching is the twin; variant aggregation is used as an unmatched control.arXiv 2405.05941 Tables IV, VNone (real-trained policies)92 cells
LIBEROFranka · –NoSim only.arXiv 2306.03310––
CALVINFranka · –NoSim only.arXiv 2112.03227––
Ranked, real robots
RoboDojo-RC– · I2RT YAMReal rebuiltRebuilds six of RoboDojo's real tasks on YAM arms with adapted rubrics. Only GR00T N1.7 is tested on both, too few to compare.github.com/robocurve/robodojo-rc––
DataCenterBench– · I2RT YAMNoReal only.robocurve.org/datacenterbench––
AA-Robotics– · Hiwonder NexArmNoReal only.artificialanalysis.ai––
ArmnetBench– · SO-101NoReal only.arXiv 2607.24481––
VLA-Replica– · SO-101, two labsReal rebuiltNo simulator. The real setup was rebuilt in a second lab and four policies rerun.arXiv 2605.20774 Table 4, App. G–20 cells, real-to-real reference
RoboChallenge– · UR5, Franka, ARX5, ALOHANoReal only.arXiv 2510.17950––
RoboArena– · Franka (DROID)NoReal only.arXiv 2506.18123––
PhAIL– · Franka FR3 (DROID)NoReal only.arXiv 2605.29710––
RoboColosseum (open and fine-tune tracks)– · Franka (DROID), two-arm YAM, SO-101, Unitree G1NoReal only: its leaderboard has no simulation track. Not AgiBot’s RoboColiseum.robocolosseum.ai––
Collected, not ranked
PolaRiSsplat replicas · Franka (DROID)Mirrored twinSix real scenes reconstructed as Gaussian splats. Per-cell sim scores are after sim co-fine-tuning; the paper reports r = 0.30 without it. Its LIBERO scores serve as an unmatched control.arXiv 2512.16881; polaris-evals.github.io dataYes: about 1k steps on sim demos from other scenes60 cells
BEHAVIORGalaxea R1 Pro · –NoThe challenge is sim only. A BEHAVIOR-1K pilot has one approximate paired result, too small to use.arXiv 2403.09227 App. G––
RoboChallenge v2– · five robotsNoReal only.RoboChallenge dataset card––
LIBERO-ProFranka · –NoSim only.arXiv 2510.03827––
Embodied Analysisunnamed two-arm suite · –NoSim only, and it does not name its tasks.embodiedanalysis.com––

C. Each simulator against real robots

Simulation benchmarkAgreement with real robots90% intervalRank correlation with real rating
Tracks real robots
RoboTwin 2.095%75–100, 28 matchups+0.60-0.01 to +0.99, 10 models
RoboColiseumrun by AgiBot, which also competes75%46–99, 39 matchups+0.53+0.09 to +0.82, 13 models
Embodied First Examcoding agents only75%40–100, 8 matchups+0.89+0.52 to +1.00, 6 models
Not yet clear
RoboLab-12077%36–100, 17 matchups+0.60-0.03 to +0.98, 9 models
MolmoSpaces75%33–100, 16 matchups+0.29-0.59 to +0.96, 7 models
CALVINpaper-reported75%25–100, 15 matchupstoo few models
SimplerEnv WidowXpaper-reported63%25–100, 15 matchups+0.37-0.45 to +0.94, 6 models
LIBERO-Pro2.063%25–96, 30 matchups+0.21-0.59 to +0.76, 7 models
RoboDojo (sim)agrees 94% with its own real arms62%33–89, 48 matchups+0.25-0.34 to +0.77, 13 models
No link found
LIBEROpaper-reported57%0–100, 10 matchups-0.35-0.82 to +0.33, 9 models
SimplerEnv Googlepaper-reported43%0–100, 14 matchups-0.31-0.82 to +0.35, 6 models

Against real-robot results from other teams. Too few shared policies to judge: RoboCasa365 (2), EBench (16), LIBERO-Plus (2) matchups. RoboTwin 2.0 has the highest agreement among simulators with enough matchups, but the differences between simulators are within noise (Appendix G). Paper-reported LIBERO and SimplerEnv Google show no positive link to other teams’ robots, although SimplerEnv ranks policies well against the setups it mirrors.

D. Model-family forecast

Families follow the index’s model type. For each real benchmark, the forecast is the index rating refitted on eight team-run simulators (RoboDojo sim, RoboTwin 2.0, RoboLab-120, MolmoSpaces, RoboCasa365, RoboColiseum, EBench and Embodied First Exam), leaving out every benchmark from the target’s team. Rating gaps convert to win probabilities with the Elo curve; observed win probabilities are Φ(gap / noise). The real-robot excess of +22 points holds when we drop any one WAM (+18 to +25) or benchmark (+18 to +34), weaken the rating prior (+15), add paper-reported suites (+25), restrict to VLAs with equally thin simulation coverage (+24), or restrict to VLAs released after the first WAM (+34). LLM coding agents beat the forecast against VLAs by similar margins in simulation and on real robots (+15 and +16 points), which places their error in the forecast rather than the transfer. The like-for-like comparison in Section 6 uses the four WAMs with forecast pairs in both domains (Cosmos3-Nano-Policy, DreamZero, InternW0-Δ, OpenWAM-α). Developers choose which checkpoints reach some real benchmarks. On RoboDojo, where the benchmark team chooses and trains every policy, WAMs win all 21 matchups against VLAs, against a forecast of 88%; on benchmarks that run developer-supplied checkpoints they win all 17, against 64%. Removing every LLM agent from every benchmark and refitting leaves the real-robot excess at +22 points. Four policies on RoboDojo (DM0, InternVLA-A1, SLA-Preview, Spirit v1.5) have no outside simulation results and drop out of its forecast.

SimulatorWAM vs. VLAVLA vs. VLA
Same team, same tasks as the real benchmark
RoboDojo (sim)against RoboDojo’s real arms100%20 of 2083%19 of 23
Other teams’ simulators
RoboTwin 2.0100%6 of 688%14 of 16
EBench100%6 of 6100%10 of 10
MolmoSpaces100%5 of 567%10 of 15
RoboColiseum100%5 of 581%13 of 16
RoboLab-12071%5 of 783%10 of 12
All five93%27 of 2983%57 of 69

Share of real-robot matchups a real benchmark decides in which the simulator’s score gap picks the real winner.

E. Design comparisons

Versions of the same benchmark on the same policies and real matchups, changing one design choice at a time. These comparisons are observational, not randomized.

ChangeComparisonResultEvidence
Mirror the real scene in simSame real results, predicted by a matched vs. an unmatched sim (SimplerEnv; PolaRiS against LIBERO)Rank correlation 0.59 → 0.85 (difference CI +0.00 to +0.71) and 0.85 → 0.91 (+0.00 to +0.19). Across sources, twins reach 0.91, against 0.49 for same-named tasksSuggestive
Run sim and real the same wayRoboDojo’s simulator vs. a second real arm, predicting a held-out arm94% vs. 97% agreement. Same verdict as RoboDojo’s own robots on 13 of 13 outside matchupsStrong
Perturb a saturated suiteLIBERO-Pro2.0, unperturbed vs. static perturbations, 7 models39% → 61% of real matchups ordered correctly (+22 points, CI 0 to +48)Suggestive
Train every policy the same wayReal benchmarks with one training recipe (ArmnetBench, VLA-Replica, RoboDojo) vs. developer-submitted checkpoints (RoboChallenge, RoboArena)Sim agreement 75% vs. 59% (+16 points, CI −19 to +48)Suggestive
Randomize scenes in a spread-out simRoboTwin 2.0, clean vs. randomized scenes, 10 models92% → 95% (+3 points, CI −10 to +14)No gain found
Run more simulation episodesHow sure the simulator is of the calls real robots reverse66% of the 64 reversed calls are at least 99% certain. More episodes would not change themNo gain expected

F. Robustness

SettingSim ↔ real agreement
As published73%
Every noise level halved72%
Every noise level doubled81%
Call threshold 80% instead of 90%73%
Call threshold 95%71%
No threshold: every matchup, including close calls68%

Without a confidence threshold, so that close calls count, sim ↔ real agreement is 68% over 442 matchups and real ↔ real 60% over 120 (difference +9 points, 90% CI −12 to +31). Leaving out one benchmark at a time keeps sim ↔ real agreement between 67% and 78%.

LLM agents. Excluding every matchup that involves an LLM agent leaves sim ↔ real agreement at 73% (224 matchups) and sim ↔ sim at 93%. Real ↔ real agreement falls from 70% to 61% over 55 matchups (90% CI 25–91%), within its wide interval. Refitting the index without the three benchmarks that test only agents (AA-Robotics, DataCenterBench and the simulated Embodied First Exam) leaves four agents in the top ten, as in the full refit. Trial shares exclude the two pairwise-rated benchmarks. Agent entries are counted per benchmark; the figure counts RoboColosseum’s two tracks and PhAIL’s two task sets separately.

G. Ranking engines and single benchmarks

The data cannot rank physics engines. Pooled by engine, simulators pick another team’s real winner 74% of the time on Isaac Sim and Isaac Lab (4 benchmarks, 78 matchups), 76% on SAPIEN (3, 34) and 66% on MuJoCo (5, 29). A permutation test finds no difference (p = .58). The spread within an engine exceeds the spread between engines: RoboTwin 2.0 and paper-reported SimplerEnv Google both run on SAPIEN and agree 95% and 43% of the time. Who runs a benchmark and how it is built matter more than its physics engine. Engines follow each benchmark’s documentation; MolmoSpaces assets also load in other simulators, and its benchmark runs in MuJoCo.

Nor can the data rank single simulators by how well they predict real robots. Each simulator decides 2 to 29 matchups against other teams’ robots. Telling 95% agreement from 70% with 80% power takes about 71 matchups per benchmark once shared policies are accounted for, and telling 85% from 70% takes about 259. Differences across the 14 simulators fall short of significance (p = .08) under a permutation test that, ignoring shared policies, already overstates them. Direct comparison fails for the same reason: only 3 of 58 simulator pairs disagree with each other on five or more of the matchups they share.

A weaker question has an answer: which benchmarks agree with the rest of the index. We refit the index without each benchmark’s team and score the benchmark’s decided matchups against the refit, using every matchup it decides rather than only those a real robot also decides. Expected agreement comes from the refit’s rating gap, so a benchmark that compares only distant policies earns no credit for it. RoboTwin 2.0 agrees 14 points more often than expected over 164 matchups; RoboChallenge agrees 48 points less often over 25. MolmoSpaces and RoboCasa365 also clear expectation on fewer matchups. Embodied First Exam and DataCenterBench clear it on 12 matchups each between coding agents rated far apart, which we do not count as evidence. Every other benchmark, real or simulated, is indistinguishable from the rest.

BenchmarkDecided matchupsAgreement with the indexPoints above expected90% interval
Embodied First Examsimulation, coding agents only12100%expected 78%+22+15 to +29
RoboTwin 2.0simulation16493%expected 78%+14+9 to +19
MolmoSpacessimulation2295%expected 81%+14+6 to +22
RoboCasa365simulation3494%expected 80%+14+2 to +22
RoboColosseum (open)real robots2488%expected 76%+12−5 to +26
CALVIN (paper-reported)simulation2085%expected 75%+10−18 to +38
DataCenterBenchreal robots, coding agents only12100%expected 94%+6+4 to +9
EBenchsimulation4286%expected 80%+5−9 to +18
RoboDojo (sim)simulation34480%expected 79%+1−11 to +11
VLA-Replicareal robots1479%expected 78%+1−40 to +23
RoboColiseumsimulation12280%expected 80%−1−18 to +14
PhAIL (DROID tasks)real robots1275%expected 76%−1−39 to +26
RoboLab-120simulation4078%expected 79%−1−23 to +18
SimplerEnv WidowX (paper-reported)simulation3171%expected 74%−3−23 to +17
PhAIL (pick and place)real robots367%expected 74%−8−61 to +29
RoboDojo (real)real robots4473%expected 81%−9−42 to +16
LIBERO (paper-reported)simulation3361%expected 72%−12−35 to +12
ArmnetBenchreal robots1567%expected 78%−12−58 to +24
RoboArenareal robots1963%expected 76%−13−60 to +23
LIBERO-Pro2.0simulation3759%expected 73%−14−45 to +11
RoboColosseum (fine-tune)real robots850%expected 66%−16−77 to +29
SimplerEnv Google (paper-reported)simulation2259%expected 76%−17−54 to +14
RoboDojo-RCreal robots2458%expected 79%−21−60 to +11
LIBERO-Plussimulation1030%expected 77%−47−86 to +11
RoboChallengereal robots, developer-submitted checkpoints, run remotely2528%expected 76%−48−74 to −15

Agreement with an index refitted without the benchmark’s team, on matchups the benchmark decides and the refit rates. Expected agreement is a logistic function of the refit’s rating gap, fitted on all benchmarks together; intervals resample policies. Agreement with the index is not agreement with real robots. The rest of the index is mostly simulation, and a benchmark that shares the simulators’ blind spots scores well here. RoboChallenge differs from the index in design as well as outcome.

H. Checks added after review

Coverage, same policies. For the 10 policies with two or more simulation and two or more real benchmarks, we refit the simulation-only and real-only ratings 200 times, each time keeping one randomly drawn benchmark of each kind for these policies (every other policy keeps all its results). The correlation falls from 0.87 to a median of 0.45, and 99% of draws fall below the full-coverage value.

Benchmark-pair features. Across all 701 decided cross-team matchups, simulated and real, agreement does not rise when the two benchmarks share a robot, both train every policy with one recipe, or score the same way. In a joint logistic fit only the pair’s kind matters: simulator–real and real–real pairs agree less than simulator–simulator pairs. The features are coarse, and the analysis shows association only.

Benchmark pairAgree when trueAgree when falseDifference, points
Same robot78%152 matchups84%549 matchups−7−22 to +4
Both train every policy with one recipe79%39 matchups83%662 matchups−3−66 to +13
Same scoring type83%612 matchups82%89 matchups+1−15 to +21
Either side paper-reported78%215 matchups85%486 matchups−7−31 to +13

WAM success rates. On benchmarks scored in percent, for the 30 WAM–VLA pairs also observed in simulation, WAMs score 32 points higher on real robots (90% CI 28–38) and 21 points higher in simulation (14–33). The real-robot figure rests on three benchmarks.

Close calls in forecasts. Hiding each policy’s real results and forecasting it from simulation alone, the index picks the real winner in 57% of the 141 matchups where the two policies are rated within 150 points of each other.

I. Limitations

  • Real-to-real agreement rests on 60 matchups (90% CI 35–100%). The matched comparison on shared matchups and the RoboDojo test carry the setup result alongside it.
  • The coverage comparison in Figure 4 rests on 10 well-covered policies, and its intervals overlap (0.64–0.99 against 0.36–0.77).
  • Counting only decided matchups favors large gaps. Twins never confidently pick the wrong winner; close calls between strong policies still need many real trials, and simulation-based forecasts for closely rated policies are little better than a coin flip (Appendix H).
  • Each twin was validated by its own builders on three to six policies per task, mostly older policies in SimplerEnv and checkpoints of a single model in Genie Sim 3.0. None of the paired datasets includes a WAM; for WAMs, same-setup evidence is RoboDojo alone, whose simulator calls all 20 matchups between its two WAMs and VLAs correctly. An independent rerun of one mirrored twin, on policies its builders did not test, is the most direct next test.
  • The model-family result rests on 7 WAMs and 38 real matchups; its size is uncertain (+12 to +35 points). Developers choose where to submit on some real benchmarks. RoboDojo trains every policy itself, and WAMs lead there too.
  • Several DROID benchmarks share policies and reference data, which makes their comparisons less independent than their count suggests. Coding agents’ results mix harness and model. Agreement between benchmarks is not ground truth.
REPRODUCE THE ANALYSIS

Data and code.

Every number on this page is generated from these files.

Evaluate your model with Playground.

Tell us about your model and what you’d like to test. We’ll help scope a custom evaluation or work with you to add your policy to the Robot Intelligence Index.