Robot Intelligence Index
Methodology

How we turn 23 public robot benchmarks into one rating per model.

LeaderboardResearch postLatest updateReport an error

At a glance

Sources
28 public leaderboards and result sets indexed, 23 of which feed the ranking.
Data
164,783 rows pulled from each leaderboard's own data, each with its source and retrieval date.
Models
102 entries matched by hand by name and version, counting 11 checkpoints trained per task on RoboChallenge as their own entries. Release dates are the first public paper, announcement, weights or listing.
Rating
Weighted Bradley–Terry fit to noise-aware head-to-head outcomes within each benchmark, on a 400-point chess-style scale centered at 1500.
Uncertainty
300 re-runs resampling benchmarks, with related simulators drawn together, and score noise. Rank ranges cover 90% of re-runs.
Checks
Our fit to RoboArena's raw comparisons reproduces its ranking at 0.985.

What the index includes

Benchmarks. We included public robot leaderboards that publish a score for every model and were active in 2026. That covers benchmarks whose own teams run or verify each entry, such as RoboDojo, RoboChallenge, RoboArena, RoboLab, AgiBot's RoboColiseum, Positronic's PhAIL, Poke & Wiggle's Reality Check, the RoboColosseum team's real-world sites, Robocurve's RoboDojo-RC and DataCenterBench, Artificial Analysis's AA-Robotics, Armnet's ArmnetBench and UT Dallas's VLA-Replica, and the benchmarks most often reported in recent model papers, such as LIBERO, SimplerEnv and CALVIN, which are self-reported. A benchmark feeds the ranking only when some of its models also appear on another benchmark. We index 28 sources, and 23 of them feed the ranking. The benchmark table explains why the other 5 do not.

Coding agents. AA-Robotics tests only general coding agents, which write code to drive a real arm with no robot training. RoboDojo-RC tests the same kind of agent alongside robot models, and Robocurve's DataCenterBench tests agents on ten steps from data-center manuals. All three enter as real-robot benchmarks. RoboColosseum's open track also runs coding agents through Robocurve's harness, on the same real robots as robot models. AA-Robotics and DataCenterBench have lower benchmark weights because they score one to three attempts per task. RoboColiseum also tests coding agents next to robot models, in simulation.

Who gets ranked. By default the overall ranking includes a model only if it has at least one real-robot result and one simulation result. 36 models meet this threshold. The coverage menu also shows any two benchmarks (64 models) or every model (102). The simulation ranking requires two simulated benchmarks for a model to be ranked, but the real-robot ranking shows every model with a real result.

Ratings. Each benchmark is turned into head-to-head comparisons between the models it tested, and one rating per model is fitted to all of them on a chess-style Elo ranking where every model starts at 1500. The methodology walks through it step by step.

Task types

Beyond the overall rating, the index rates models on four types of task. We sorted each benchmark's own task and category scores into these types by hand, assigning each task to the type that best describes its main difficulty. Each type is then rated from those scores alone. A model's score in a task type is the plain average of its scores on that benchmark's tasks of that type. A model is ranked in a task type only if it meets the overall rule and has at least two benchmarks in that type.

Task typeWhat it measuresExample tasks
Pick and placeGrasping objects and putting them in a target place or arrangement, usually in a few steps.Stack bowls (RoboDojo, RoboChallenge, RoboDojo-RC). Sort books and place shoes on a rack (RoboChallenge). Place a toy in a marked rectangle (AA-Robotics). Move household objects between bins (PhAIL). Sort screws into bins (Poke & Wiggle). RoboLab's simple tasks and RoboColiseum's general manipulation track.
Household and long-horizonChores that chain several steps, objects or tools, where an early mistake can sink the whole task.Make a vegetarian sandwich and wipe the table (RoboChallenge). Fold clothes and make toast (RoboDojo simulation). Pack and pour fruit (RoboDojo-RC). RoboCasa365's kitchen tasks and RoboLab's complex tasks.
Precision and dexterityTight tolerances, careful force or exact positioning.Insert a key and fasten screws (RoboDojo simulation). Cap a pen and insert a charger (RoboDojo real). Plug in a network cable and scan a QR code (RoboChallenge). Pour a drink into a cup (AA-Robotics). Connect a DC jack and route a cable through closed hoops (Poke & Wiggle).
RobustnessFamiliar tasks after the scene, camera, lighting, layout or wording changes.Randomized scenes (RoboDojo simulation, RoboTwin 2.0). Perturbation suites (LIBERO-Plus, LIBERO-Pro2.0). Unseen multi-step tasks (RoboCasa365). Vague instructions (RoboLab). Objects placed outside the training range (Poke & Wiggle). RoboColiseum's robustness track.

Benchmarks that report no breakdown count only toward the overall views, except where the whole benchmark tests robustness. EBench's held-out test and the perturbation scores of LIBERO-Plus and LIBERO-Pro2.0 also count toward robustness, as does VLA-Replica's out-of-distribution score. DataCenterBench counts toward precision only, because each of its tasks is one fine control or connector step. CALVIN's average chain length, from 0 to 5, is multiplied by 20 to put it on the same 0 to 100 scale as other scores. RoboArena counts toward pick and place because pick and place tasks dominate its task breakdown.

Indexed but not ranked

We index 28 sources, and 23 of them feed the ranking. The other 5 are collected with every update but do not feed it, for the reason given.

SourceTypeReason
PolaRiSReal-to-simPublishes correlations only, no per-model scores.
BEHAVIOR Challenge 2025SimulationTeam entries share no models with any other board.
RoboChallenge Table30 V2, CVPR and ICRA tracksReal robotsOnly 2 entries cover all V2 tasks, and CVPR scores do not match the public runs.
LIBERO-PRO (original)SimulationSuperseded by LIBERO-Pro2.0 from the same team.
Embodied Analysis, Physical Agent Capability IndexSimulationPublishes only an Elo score per model. It does not name its simulation tasks or publish comparison counts or uncertainty.

How ratings are computed

Each model gets one rating, built from head-to-head comparisons inside every benchmark it appears on. Every model starts at a score of 1500, and the gap between two ratings predicts how often one outscores the other.

1. Collect each benchmark's scores

We pulled every benchmark's public results as of November 17, 2026, and took one headline score per model from each. The table at the end of this section lists the score used for each benchmark. We matched the same model across benchmarks by hand, using name and version only. We leave out entries without a version or with an ambiguous name. A policy wrapped in a separate planner is not merged with the bare policy, and neither is a model trained on a benchmark's own data under a new name, such as π0.5-MolmoSpaces. Under this framework, a model fine-tuned separately for two benchmarks still counts as one model. A separate checkpoint trained for each task of one benchmark is different. We list those checkpoints as their own entry, marked per-task, so they do not stand in for the model they fine-tune. We discuss both choices in the limitations section.

Where a benchmark publishes success against time, we use its success at the longest time limit it reports, so a rating reflects whether a model finishes a task rather than how fast. Only PhAIL publishes this. Its DROID headline counts how much of each task is done within 240 s, and its success curves run to 480 s. We add each model's gain between the two on PhAIL's own curve to its headline score. Only GPT-6 Astra ran past 240 s, and it rises from 55.9 to 68.5, from fifth of seven to first. The other six had nearly stopped improving, gaining at most 1.7 points between 180 and 240 s. The noise on PhAIL's 480 s scores keeps the width of its published 240 s range.

RoboArena and RoboColosseum's open track publish ratings rather than success rates. We use each rating as the score and its published uncertainty as the noise, and count a RoboArena entry only once it has at least 20 evaluations. RoboColosseum rates each robot separately, so each robot is its own table, and its SO-101 table counts 0.71 times as much as the others because its ranking held up in only 59% of the benchmark's site splits, against about 85% on the other robots. When a benchmark re-runs a model, the new result replaces the old one. When a source cannot be reached, the index keeps the last data it retrieved.

2. Turn each benchmark into head-to-head matchups

Within a benchmark, every pair of models it tested is one matchup. Instead of a plain win or loss, each matchup records how likely it is that one model really scored higher, given how noisy that benchmark is.

qAB= Φ( sA−sB σA2+σB2 )

qAB is the probability assigned to A outscoring B, sA and sB are their benchmark scores, and σA and σB are the assumed standard deviations of their score noise. Φ is the standard normal cumulative distribution function. Noise describes how much a score could move if the benchmark were run again. On RoboDojo's real robots, π0.5 scores 22.9 and X-VLA 7.6, each with a noise of 2.5 points. The gap is about four times the combined noise, so π0.5 counts as better with near certainty. GR00T N1.7 at 5.9 against π0 at 5.8 counts as 0.51, almost a tie.

Treating the two scores' noise as independent ignores that models tested on the same tasks tend to succeed and fail together. On a shared task set that usually overstates the noise in their difference, which makes matchups softer rather than more decisive, though other patterns of dependence can work the other way. For most benchmarks the noise is the sampling error plus a fixed 1 to 3 points, so a tiny gap between two precisely measured scores does not become a near-certain win. Where an entry has no published interval, RoboLab-120's noise is set to 2 points, and where MolmoSpaces gives no episode count we assume 1,000 episodes. Task-type scores use the same approach, with the sampling error computed from the attempts behind each average.

3. Weight each benchmark

In our index, each benchmark has a benchmark weight, which is how independent its evaluators are multiplied by how repeatable its scores are. We judge and score independence by rubric, and we take repeatability from published repeat runs where they exist and from attempts per task where they do not. Each matchup is scaled by the size of its benchmark to ensure a model’s result counts for roughly the same amount regardless of how many models are included in a benchmark's leaderboard. Benchmarks that publish several separate tables, such as MolmoSpaces' tasks or Poke & Wiggle's separate tables for robot models and agents, split their benchmark weight across those tables, and a model that appears in only one table gets only that table's share.

We set both factors with a fixed rubric. The evaluator factor is 1.0 when the benchmark's own team or an independent operator runs or verifies every entry. It is 0.9 when an independent party runs every entry but the evaluation has a known limit, such as volunteer labs running the comparisons, scoring without blinding, or an operator that fine-tunes the models it tests. It is 0.85 when a competing model's authors help run the benchmark, 0.8 when the operator runs only part of the evaluation, results mix operator runs and submissions, or the operator enters its own model, 0.75 when submitters run their own models and the operator also builds models, 0.7 for results posted in a benchmark's README and 0.4 for numbers self-reported in papers.

The repeatability factor comes from published reliability measurements for RoboChallenge (0.70, its repeat-run correlation) and RoboArena (0.90, its split-half agreement). For the other benchmarks it follows from how many attempts each runs per task and how they are scored, from 0.95 for the largest evaluations down to 0.55 for DataCenterBench, which scores three tries on each of ten tasks with LLM judges. The table below gives both factors for every benchmark. Because these judgments were subjective, we also tested robustness to several different weighting schemes, and found broad agreement between them. Section 9 reports those results in detail.

BenchmarkEvaluatorRepeatabilityBasis
RoboDojo (real)1.00.90Run by RoboDojo's team, 10 trials per task, blinded raters
RoboDojo-RC Tier 11.00.90Independent operator (Robocurve), 120 attempts per model
AA-Robotics1.00.60Independent operator (Artificial Analysis), one human-judged attempt per task
DataCenterBench Tier 11.00.55Independent operator on the RoboDojo-RC robots, three tries on each of ten tasks, judged by LLMs with a human tie-break
PhAIL1.00.85Operator-run, every episode published on video, about one attempt per DROID task
RoboArena0.90.90Double-blind comparisons run by volunteer labs; split-half agreement 0.90
ArmnetBench0.90.85Third-party arm farm, about 360 rollouts per model, scored by the operator without blinding
VLA-Replica0.90.80Independent lab, 15 trials per task, reproduced at a second site
Poke & Wiggle Reality Check0.90.90Operator-run and rated by people, every episode on video; the operator sells evaluations and fine-tunes the robot models it tests; one sitting per model
RoboColosseum, open track0.850.85Blind comparisons on new tasks at many sites; MolmoAct2's authors help run it
RoboColosseum, fine-tune track0.850.80Operator-run, 25 trials per task; MolmoAct2's authors help run it
RoboChallenge0.80.70Operator's robots, with each submitter running its own model; repeat runs correlate at 0.70
RoboDojo (simulation)1.00.95Run by RoboDojo's team, 150 episodes per task
RoboLab-1201.00.95Run or cross-checked by the operator, 1,200 episodes
Embodied First Exam1.00.95Operator-run, 456 standard tests
LIBERO-Pro2.01.00.85One team runs every model's official checkpoint
RoboTwin 2.00.80.90Mix of operator runs and submissions, 100 trials per task
MolmoSpaces0.80.90Mix of operator runs and submissions, about 1,000 episodes per task
RoboCasa3650.80.90Submitted, then verified by the operator, 50 rollouts per task
RoboColiseum0.80.85Run by AgiBot, which also enters its own model; episodes per task not published
EBench0.750.85Held-out test on the operator's server, with each submitter running its own model; the operator also builds models
LIBERO-Plus0.70.85Results posted in the benchmark's README
Paper-reported0.40.70Self-reported in papers

Simulation results get two further adjustments, adopted on November 11 from our sim-to-real study. In the overall and task-type rankings every simulation benchmark counts half as much as it otherwise would, because simulators share errors and are not independent votes. Each simulator is also scaled by an agreement factor, from 0.88 to 1.17 today, that reflects how often it picks the same winner as other teams' real robots. The simulation ranking uses the agreement factors but not the halving. In the task-type rankings, paper-reported results keep a factor of 1. The sim-to-real agreement tables below list both.

4. Fit one rating per model

Our model uses two ratings to predict a matchup's outcome:

pAB= 11+ 10− RA−RB400

pAB is the predicted probability that A outscores B, and RA and RB are their fitted ratings.

We choose the set of ratings whose predictions best match all the weighted matchup outcomes at once. This is a weighted, regularized Bradley–Terry model fitted to the probabilistic matchups from step 2. Fitting Bradley–Terry ratings to results pooled across many benchmarks has precedent in machine learning, from comparing algorithms across data sets to ranking agents across Kaggle competitions.

Models that never met are placed through the models they share, with shared models acting as bridges between benchmarks. A small pull toward 1500 keeps a model with little evidence from landing at an extreme. The pull works like a normal prior centered on 1500 with a standard deviation of about 550 rating points. We fit the overall, simulation, real-robot and four task-type rankings separately, so a rating only compares models within one setting.

5. Read a rating

A gap between two ratings gives the predicted chance that the higher-rated model outscores the other.

Rating gapPredicted chance the higher-rated model outscores the other
050% of the time
10064%
20076%
40091%

The table above follows from the formula in step 4. We checked these predictions against unseen results, refitting the ratings 23 times, leaving out one benchmark each time, and predicting the order of every pair of models that benchmark tested with different scores, with 1,896 pairs in all. We find the fitted model's predictions strongly correlate with real outcomes. When the model gave the higher-rated model a 90 to 100% chance, that model outscored the other 97% of the time. In the 70 to 80% range it did so 83% of the time, meaning predictions in this band understate the winning model's relative strength. These checks only cover benchmarks like the ones in the index. Relative performance may vary on tasks that are out of distribution.

6. Measure uncertainty

We re-ran the whole fit 300 times. Each re-run draws benchmarks at random with replacement and nudges every score by its noise. A benchmark drawn more than once still counts once, so each re-run keeps about two thirds of the benchmarks and leaves out the rest. Related simulators are drawn together as one unit, the LIBERO family (LIBERO-Pro2.0, LIBERO-Plus and paper-reported results) and the two DROID-Franka simulators (RoboLab-120 and MolmoSpaces), so a blind spot they share cannot count twice. A model's rank range is where its rank falls in 90% of those re-runs, from the 5th to the 95th percentile, rounded down. In each re-run a model is ranked only against the models that re-run still includes. The leaderboard sorts by rating. Sorting by worst plausible rank instead uses the bottom of that range, which penalizes models with uncertain rankings. A model's range comes from the re-runs that include its benchmarks.

A model with only one benchmark in a leaderboard shows no range, since no re-run can test it against a second. When a re-run drops a benchmark that links two groups of models, the groups are placed relative to each other only by the pull toward 1500. In 2,000 separate draws, that happens in under 1% of re-runs overall, 2% in simulation and 3% of real-robot re-runs. Twelve real-robot benchmarks are in the final set. RoboDojo-RC and RoboColosseum's open track test coding agents and robot models on the same robots, and seven of RoboColosseum's models also appear on other benchmarks in the index. Real-robot rank ranges are therefore less reliable than the others.

7. Measure disagreement

For each benchmark in a view, we place the model by percentile among the models we matched there, from 100 for first to 0 for last. Where a benchmark has several tables in the view, its percentile is the average across them. We calculate spread as the sample standard deviation of those percentiles. The leaderboard shows it as ± N under each rating. It measures how much benchmarks disagree about a model, not the uncertainty in its rating, which the rank range covers. Dream-1 finishes first on RoboDojo-RC and in the 28th percentile on RoboColiseum, for a spread of 51. Claude Fable 5.2 finishes in the 91st percentile or higher on all four of its benchmarks, for a spread of 4. A model with one benchmark in the view has no spread. Because percentiles depend on which other models each benchmark tested, spread reflects differences in the level of competition as well as disagreement between benchmarks.

8. Decide who gets a rank

Every model's comparisons inform the full set of ratings, but only some models receive a rank. By default, the overall ranking requires at least one real-robot and one simulation result, the simulation ranking requires two simulated benchmarks, the real-robot ranking includes every model with a real result and the task types follow the overall rule while also requiring two benchmarks within the task type. We hold back entries whose underlying model or agent harness is undisclosed, or that were scored only after using several times the time other entries on the same benchmark were given, until their developers document them. AA-Robotics scores for GPT-6.2 Astra, GPT-6.2 Sol, Claude Fable 5.2, Gemini 4 Hydrogen and Gemini 4 Carbon, published on November 17 as a chart with no harness, effort setting or task scores, are held this way until Artificial Analysis publishes the runs. We also hold back entries that a benchmark itself marks as not following its rules, such as RoboDojo agent-track entries marked with an asterisk. An entry the benchmark also scores at the time limit the others had is not held back; step 1 explains which limit we use. Held entries stay in each benchmark's table but do not enter the fit. We make one exception. Claude Fable 5.2's RoboDojo simulation score comes from Anthropic's own runs in a harness it has not published, and RoboDojo has not listed it. We count it like RoboDojo's own entries and tag it as self-reported in the leaderboard.

9. Check robustness

We refit the overall ranking under alternatives to our input values for benchmark weights, the weight given to simulation, noise and the prior. The table compares every alternative with the published top five, Claude Fable 5.2, SLA-Preview, Dream-1, GPT-6 Astra and GPT-6.1 Sol.

AlternativeFirst placeTop five
Equal benchmark weights, simulation at full weight and no agreement factorsClaude Fable 5.24 of 5, with InternW0-Δ replacing Dream-1
Simulation at full weight, as before November 11Claude Fable 5.24 of 5, with InternW0-Δ replacing Dream-1
Simulation at a third, or three quarters, of a real-robot resultClaude Fable 5.2 in both4 of 5 in both, with InternW0-Δ replacing GPT-6.1 Sol at a third and Dream-1 at three quarters
No agreement factorsClaude Fable 5.2Same five, same order
Evaluator factor only, or repeatability factor onlyClaude Fable 5.2Same five in both, reordered with the repeatability factor only
Each benchmark weight scaled by a random factor between ½ and 2, drawn evenly on a log scale, 500 timesClaude Fable 5.2 in every draw4.3 of 5 kept on average, all five in 36%
Each benchmark weight scaled by a random factor between ¼ and 4, drawn evenly on a log scale, 500 timesClaude Fable 5.2 in 97%, SLA-Preview in 3%, Dream-1 in the rest3.8 of 5 kept on average
Each of the 23 benchmarks left out in turnClaude Fable 5.2 in all 23Same five in 13 of 23. 4 of 5 in the other 10, with InternW0-Δ entering in 9 of them
Paper-reported results left outClaude Fable 5.2Same five, same order
LIBERO family left out (LIBERO-Pro2.0, LIBERO-Plus and paper-reported)Claude Fable 5.2Same five, same order
DROID-setup benchmarks left out (RoboArena and PhAIL)Claude Fable 5.2Same five, same order. Cosmos3-Nano-Policy loses its only real-robot result
SO-101 benchmarks left out (ArmnetBench and VLA-Replica)Claude Fable 5.2Same five, same order
RoboColosseum left out (both tracks)Claude Fable 5.2Same five, same order
RoboDojo family left out (simulation, real robots and RoboDojo-RC)Claude Fable 5.23 of 5. SLA-Preview loses its only real-robot result and Dream-1 loses RoboDojo-RC, and Cosmos3-Nano-Policy and Claude Opus 5.5 enter
All noise halved, or all noise doubledClaude Fable 5.2 in bothSame five, same order, when halved. 3 of 5 when doubled, with InternW0-Δ and MWA-Phoenix replacing GPT-6 Astra and GPT-6.1 Sol
Pull toward 1500 five times weaker, or four times strongerClaude Fable 5.2 in both4 of 5 in both, with InternW0-Δ replacing GPT-6 Astra when weaker and Dream-1 when stronger

Claude Fable 5.2's first place holds by a wide margin in every alternative, including leaving out RoboDojo-RC, Poke & Wiggle or its self-reported RoboDojo simulation score, and leaving out the whole RoboDojo family. Places two to nine depend more on these choices. SLA-Preview's second place rests on RoboDojo's real-robot track, its only real-robot result, which it leads by 3.4 points, and it stays second or third in 90% of the random reweightings between ½ and 2. Dream-1's place rests on the higher weight for real-robot results. It leads RoboDojo-RC, its only real-robot result, and finishes 24th of 30 on RoboColiseum, its only simulation result, so it leaves the top five when simulation counts three quarters as much as a real result or more. RoboDojo's real-robot track gives SLA-Preview, InternW0-Δ, MWA-Phoenix and OpenWAM-α their only real-robot result, and RoboColiseum gives Dream-1 its only simulation result. PhAIL gives FLUX 3 Action and Cosmos3-Nano-Policy their only real-robot result.

Benchmark weights and noise

BenchmarkScore usedBenchmark weightNoise per scoreWeight in overall ranking
RoboDojo (real)Overall progress score0.902.5 points0.90
RoboDojo-RC Tier 1Mean progress over 120 attempts0.90From its spread across attempts, about 3 points0.90
AA-RoboticsMean task score over 8 tasks0.60About 14 points, one attempt per task0.60
DataCenterBench Tier 1Mean score over 10 tasks, three tries each0.55Its published 95% range0.55
ArmnetBenchStrict success over 12 tasks0.77Sampling error over about 360 rollouts, plus 20.77
VLA-ReplicaIn-distribution success over 10 tasks0.72Sampling error over 150 trials, plus 20.72
RoboChallengeProgress over 30 tasks0.56About 3.7 points, from its repeat runs0.56
RoboArenaHead-to-head rating0.81Its published uncertainty0.81
RoboColosseum, open trackRating on each robot, weight split0.72Its published uncertainty0.72
RoboColosseum, fine-tune trackProgress over 20 tasks, by robot0.68Sampling error over 25 trials per task, plus 20.68
PhAILReliability on 286 DROID tasks, at 480 s, and on pick and place, weight split0.85Its published 95% range0.85
Poke & Wiggle Reality CheckSuccess over 10 tasks, robot models and agents in separate tables, weight split0.81Sampling error over 5,400 or 900 episodes, plus 20.81
RoboDojo (simulation)Overall progress score0.95Sampling error over 6,300 episodes, plus 1 point0.42
RoboTwin 2.0Average of clean and randomized0.723 points0.42
RoboLab-120Success rate0.95Its published 95% interval0.51
MolmoSpacesSuccess per task, weight split0.72Sampling error over about 1,000 episodes, plus 30.36
RoboCasa365Overall success rate0.72Sampling error over about 900 episodes, plus 30.36
RoboColiseumTotal score over 38 tasks0.68Sampling error assuming about 20 episodes per task, plus 20.34
EBenchSuccess on the held-out test0.64Sampling error over 794 instances, plus 20.35
LIBERO-Pro2.0Success under perturbations0.852 points0.39
LIBERO-PlusSuccess under perturbations0.603 points0.30
Embodied First ExamShare of 456 standard tests solved0.95Sampling error over 456 tests, plus 10.51
Paper-reportedLIBERO, LIBERO-Long, SimplerEnv WidowX and Google Robot, CALVIN and RoboCasa, weight split six ways0.284 points0.12–0.14

Benchmark weight is the evaluator factor multiplied by the repeatability factor, both listed in step 3. The last column is the benchmark's weight in the overall ranking, after the halving and agreement factor for simulators. Giving every benchmark equal weight, with simulation at full weight and no agreement factors, keeps four of the top five, with InternW0-Δ in place of Dream-1, and rank correlation 0.97 overall and 0.90 or higher in every other leaderboard. Section 9 reports wider changes to the weights.

Sim-to-real agreement

Our sim-to-real study found that simulators from different teams often share the same mistakes, and that a simulator picks the same winner as another team's real robot about as often as two real-robot benchmarks from different teams agree. We publish these rates with every update, recomputed on the index's current data. A benchmark decides a matchup when its score gap would survive a rerun at least 90% of the time under the noise in step 2, and two benchmarks agree when both decide a matchup and pick the same winner. We never compare two benchmarks run by the same team, and we split paper-reported results by suite.

ComparisonSame winnerMatchups
Simulators against other teams' real robots80%229
Real-robot benchmarks from different teams82%50
Simulators from different teams86%374
SimulatorAgrees with other teams' real robotsMatchupsAgreement factor
RoboDojo (simulation)70%400.88
RoboColiseum81%321.01
RoboTwin 2.097%311.17
LIBERO-Pro2.070%230.92
Embodied First Exam89%191.07
EBench100%141.11
RoboLab-12093%141.07
LIBERO, paper-reported73%110.97
RoboCasa, paper-reported55%110.89
CALVIN, paper-reported70%100.96
SimplerEnv Google Robot, paper-reported62%81 (too few)
SimplerEnv WidowX, paper-reported62%81 (too few)
MolmoSpaces100%51 (too few)
RoboCasa365100%21 (too few)
LIBERO-Plus100%11 (too few)
Model families in the matchupSimulator agrees with real robotMatchups
VLA against VLA74%138
LLM against LLM86%28
VLA against Other95%22
WAM against VLA100%21
VLA against LLM100%15
WAM against LLM0%4
WAM against WAM0%1

The agreement factor pulls each simulator's agreement toward the pooled rate, as if it had 71 more matchups at that rate, then rescales it so a simulator at the pooled rate gets 1 and one at chance gets 0, within limits of 0.25 and 1.5. Simulators with fewer than 10 decided matchups keep a factor of 1. The differences between simulators are still within noise, so the factors stay close to 1 for now and will spread as shared real-robot results accumulate. Listing per-task checkpoints separately raised overall agreement from 75% to 79% at the time.

Cost and time

The leaderboard, benchmark and model pages show what one task attempt costs and how long it takes. Few benchmarks publish either, so most of these figures are Playground estimates, tagged est. in the tables. They cover the compute to run a model, not the robot, its operators or the evaluation, and they play no part in the rating.

Time per attempt

SourceResults
Published by the benchmark, or derived from its own run data81 of 273
Estimated from the benchmark’s time limit as limit × (1 − success ÷ 2), so a model that always succeeds is assumed to finish in half the limit and one that never does to use all of it. Benchmarks that publish ratings rather than success rates use 50%.176 of 273
The median measured time on the same benchmark, where it publishes no time limit16 of 273

Time limits on simulated benchmarks are in simulated seconds, so simulated times describe the length of an episode rather than the computing time to run it.

Cost per attempt

SourceResults
Published by the benchmark32 of 273
A benchmark’s published total spread over its rated trials, or, where a model has none, another model’s published cost on the same benchmark scaled by API price6 of 273
For coding agents without a published cost, the agent’s own median published cost per attempt on other benchmarks, or the median across agents when it has none19 of 273
For robot models served through a paid API, the provider’s price per robot-hour × time2 of 273
For other robot models, one H100 at $3.29 per hour (Lambda on-demand, Oct 2026) × time214 of 273

A coding agent’s cost is what its API calls cost, while a robot model’s is the price of renting the hardware it runs on for the length of the attempt. The two are different kinds of cost, so the rating-against-cost charts compare agents and robot models only roughly.

In the leaderboard

A model’s cost and time in a leaderboard view are the median across the benchmarks in that view, which is the average of the two middle values when there is an even number. They are tagged est. when any benchmark behind them is estimated. Model pages give the figure for each benchmark and the same median across all of the model’s benchmarks, and each benchmark page lists the figure for every model it tested.

Updates, corrections and disclosures

Updates

We check every source two to three times a day and publish an update whenever a result changes. Each update refits every ranking, rank range and robustness check, and recomputes the sim-to-real agreement factors, so a simulator's weight can shift slightly from one update to the next. The changelog records what each update added, and we keep a copy of the leaderboard from every update.

Corrections

When a benchmark corrects or re-runs a result, the new result replaces the old one in the next update, and the changelog notes the change. To report an error, email contact@withplayground.ai with the model, the benchmark and a link to the source. We check every report against the benchmark's published data. A confirmed error is fixed in the next update and noted in the changelog.

Independence and disclosures

Evaluation runs are conducted independently of Playground. Every score in the index comes from a benchmark's own published results or, where marked, a developer's self-reported results. Playground also offers evaluations to model developers, and we disclose all of Playground's relationships with model providers.

Limitations

Why combine benchmarks

No single robot benchmark tests more than a fraction of today's policies, and each covers a different mix of robots, tasks and evaluation protocols. Combining them gives the broadest comparison currently available. A model's rating reflects how it performs against many others across many settings, rather than on one task suite chosen by one team. The method is built to show where that comparison is strong and where it is thin, through rank ranges, spread and coverage thresholds.

Benchmark-specific checkpoints

Some policies are fine-tuned for a specific benchmark before evaluation. A rating therefore describes a model as its developers or each benchmark's team adapted and ran it. That matches how these policies are used in practice, but it means a rating cannot separate a model's design from the training it received for each benchmark. The spread column makes cross-benchmark disagreement visible, and helps surface cases where models were optimized to max out a specific eval suite. The exception is a separate checkpoint trained for each task of one benchmark, as some developers entered on RoboChallenge. We list those as their own entries, so CogACT, OpenVLA-OFT and RDT, whose only real-robot result was a per-task entry, are not ranked overall until another real-robot benchmark tests them.

Hardware and harness differences

Benchmarks run on different robots and give coding agents different tools. Real-robot results come from setups as different as a one-arm Hiwonder NexArm on AA-Robotics, the two-arm I2RT YAM on RoboDojo-RC and DataCenterBench, low-cost SO-101 arms on ArmnetBench and VLA-Replica, two-arm ARX X5 and Piper robots on RoboDojo, Franka arms on RoboArena and PhAIL, two-arm Franka FR3 stations on Poke & Wiggle, and Franka, two-arm YAM, SO-101 and Unitree G1 robots at everyday sites such as offices and parks on RoboColosseum. Coding agents also run inside different harnesses. Embodied First Exam pairs each model with an agent harness, such as Codex for GPT-6 Astra, Antigravity for Gemini 4 Argon and Claude Code for Claude Opus 5.5. RoboDojo-RC instead runs its LLMs with a shared system prompt that includes rig calibration, grasping advice and a 40-call limit. RoboColosseum runs them through Robocurve's harness at medium reasoning effort, with up to 30 model calls per trial. Poke & Wiggle has each agent write robot code on its own skill library from ten demonstrations, with up to 30 simulated and 10 real practice trials before the rated ones. A single rating folds these differences together, so it describes a model across the mix of robots and harnesses its benchmarks used, not on any one setup. Because every comparison happens inside one benchmark, a setup that is harder for every model does not change the ranking. Differences matter only when a model suits one setup more than another. RoboDojo measures this directly by scoring the same 13 policies on three arms, each with its own six tasks. The same four policies lead on every arm, though a different one ranks first on each, and rank correlation between arms runs from 0.76 to 0.82.

Overlap between benchmarks

Ratings are comparable only through models that appear on more than one benchmark. Every benchmark in the index shares models with at least one other, and the fit uses each shared model as a bridge. Some connections rest on very few models. Coding agents meet robot models on five benchmarks. Among real-robot benchmarks, RoboColosseum's open track puts three coding agents and five robot models on the same robots, and only GR00T N1.7 and MolmoAct2 link RoboDojo-RC to the rest. This means a shift in how one such model was prepared can move a whole group of models, which rank ranges only partially capture.

Which models each benchmark tests

Developers choose where to submit, so newer models tend to appear on fewer benchmarks. Some of the newest VLAs, including Awomo-0.5I, Axiom-0 and Xiaomi-Robotics-1, have only simulation results so far and will join the default overall ranking once a real-robot benchmark tests them. Until then they appear when the coverage menu includes every model, and the worst-plausible-rank sort favors models whose position holds up across benchmarks.

Benchmark design

Benchmarks emphasize different skills. RoboDojo-RC chose its tasks after trying GPT-6 Astra, and half of RoboColiseum's total score comes from instruction following and spatial reasoning, where coding agents do well. Benchmark weights, set by independence and repeatability, limit how much any one design can move the ratings. Giving every benchmark equal weight produces nearly the same order, with rank correlation 0.97 overall and 0.90 or higher in every task-type leaderboard, and the task-type leaderboards show how models compare on specific kinds of manipulation.

Operators who also compete

AgiBot runs RoboColiseum and also trained GO-3 Nano Preview, second in our simulation ranking. Every RoboColiseum entry is scored by the same automated evaluation. We give the benchmark a lower evaluator factor (0.8) to account for the overlap. MolmoAct2's authors help run RoboColosseum, where MolmoAct2 competes. Its open-track evaluators do not know which models they are comparing, and we give it a lower evaluator factor (0.85) too.

Judgment calls

The weights, noise levels, task sorting and model matching are judgment calls. We've documented our process for assigning these values in the methodology. We further test them two ways. First, applied to RoboArena’s raw comparisons alone, our fitting method recovers RoboArena’s own ranking (rank correlation 0.985). Second, section 9 refits the combined ranking under alternative weights, noise levels and benchmark sets, and finds little change in relative rank. The half weight for simulation is a judgment call too. At a third, and at three quarters, Claude Fable 5.2 stays first and four of the top five stay.

What comes next

As more benchmarks publish results and more models appear on several of them, the overlap behind these ratings will grow and rank ranges should narrow. We will update the index as new results appear, and publish its sim-to-real agreement with each update.

Changelog

The changelog tracks changes in benchmark methodology, new benchmarks added to the index, and new models added by existing benchmarks. We preserve historical copies of the leaderboard with each update. Please contact us directly for historical leaderboard data.

  • updated AA-Robotics

    Artificial Analysis charted new scores for GPT-6.2 Astra (79), GPT-6.2 Sol (78), Claude Fable 5.2 (75), Gemini 4 Hydrogen (58) and Gemini 4 Carbon (43) without the harness, effort setting or task scores it published for its first runs. We briefly counted them, then held them until Artificial Analysis publishes the runs. Ratings are as on November 16.

    Real robots

  • added 1 model to RoboDojo sim

    • Era0-PreviewnewNot yet ranked
    Simulation
  • added 2 models to Poke & Wiggle Reality Check

    • Mistral Large 4newReal robots

    Also covers Gemini 4 Carbon, which was already ranked.

    Real robots

  • added 1 model to RoboTwin 2.0

    • SLA-PreviewOverall
    Simulation
  • added 1 model to Embodied First Exam

    • GPT-6.1 SolSimulation · Robustness
    Simulation
  • updated RoboDojo sim

    Meituan-Robotics-0 was re-evaluated by its team, from 14.95 to 21.68. The new run replaces its earlier result.

    Simulation
  • updated PhAIL

    Some DROID episodes were re-graded. Scores moved by at most 0.2 points. The index now counts DROID tasks at 480 s, the longest limit PhAIL publishes, rather than its 240 s headline, so the rating measures whether a model finishes a task, not how fast. Only GPT-6 Astra ran past 240 s; its DROID value rises from 55.9 to 68.5.

    Real robots

  • added Poke & Wiggle Reality Check

    • Claude Fable 5.2Robustness
    • DiT-FlownewReal robots
    • RhonewReal robots

    Also covers Claude Opus 5.5, Gemini 4 Argon, GPT-6 Astra, GR00T N1.7, MolmoAct2 and π0.5, which were already ranked.

    Real robots
  • updated RoboColiseum

    FF Omnimind reran general manipulation, raising its total score from 0.902 to 0.946.

    Simulation

  • added 1 model to PhAIL

    Also covers GPT-6 Astra, which was already ranked.

    Real robots
  • added 1 model to Embodied First Exam

    • Claude Fable 5.2Simulation · Pick and place · Household · Precision
    Simulation
  • updated Embodied First Exam

    Codex + GPT-6 Astra solves 157 of 456 standard tests after reruns, up from 153. The exam keeps each agent's latest finished attempt per test.

    Simulation
  • updated PhAIL

    DROID adds a November sitting on 109 tasks run beside four board models. GPT-6 Astra joins at 55.9. π0.5 and GR00T were rerun after serving fixes and now count only their new runs, π0.5 rising from 12.0 to 62.5 and GR00T from 36.0 to 62.3.

    Real robots

  • updated the methodology

    Adopted the four changes from our sim-to-real study. Simulation results now count half as much as real-robot results in the overall and task-type rankings, and each simulator is also weighted by how often it agrees with other teams' real robots. Rank ranges resample the LIBERO family and the two DROID-Franka simulators together. Checkpoints trained separately for each RoboChallenge task are listed as their own entries, so CogACT, OpenVLA-OFT and RDT leave the overall ranking until another real-robot benchmark tests them. The index's sim-to-real agreement is now published with every update.

  • added 1 model to RoboDojo sim

    • Claude Fable 5.2newOverall
    Simulation
  • added 1 model to RoboDojo-RC

    • Claude Fable 5.2newOverall · Real robots
    Real robots
  • added 1 model to RoboLab

    • Praxis-1newNot yet ranked
    Simulation
  • added 1 model to LIBERO-Pro2.0

    • CloudRobo-WM 1.0newNot yet ranked
    Simulation
  • updated LIBERO-Pro2.0

    CloudRobo-WM 1.0 reports success rates from its own evaluation run. The benchmark's other entries come from official checkpoints run by its authors.

    Simulation
  • updated RoboDojo-RC

    Claude Fable 5.2 comes from Robocurve's own runs with the main board's tasks and judge. Robocurve has not yet put it on its main leaderboard.

    Real robots
  • updated RoboDojo sim

    Claude Fable 5.2's 64.9 is self-reported by Anthropic from its own harness. RoboDojo has not listed it, and we count it as a RoboDojo simulation result.

    Simulation

  • added 1 model to RoboDojo-RC

    • Gemini 4 CarbonnewReal robots
    Real robots
  • updated RoboDojo-RC

    Gemini 4 Carbon, an early-access Google model, now appears on Robocurve's main leaderboard.

    Real robots
  • updated RoboColiseum

    FF Omnimind reran its robustness dimension. The new run replaces its earlier result. Two new entrants, MXI SimWorld v0 Preview and HexaShell, are listed but not yet counted until their developers document the model and harness they run, since each used several times the evaluation time of other entries.

    Simulation

  • added 1 model to EBench

    • Motubrain 2.0newNot yet ranked
    Simulation
  • updated RoboColiseum

    FF Omnimind reran two of its four dimensions. The new run replaces its earlier result.

    Simulation

  • added 3 models to RoboColiseum

    • GigaBrain-0.7Simulation
    • LingBot-VLA v2Overall · Pick and place
    • FF OmnimindnewNot yet ranked
    Simulation
  • added 1 model to Embodied First Exam

    • Kimi K3.1 PronewNot yet ranked
    Simulation
  • updated Embodied First Exam

    The core view was removed, so we now use the standard view over the same 456 tests. Re-runs replaced several earlier results.

    Simulation
  • updated RoboLab

    New submissions are paused while the team revises its submission process.

    Simulation

  • updated PhAIL

    DROID tasks rescored over the 278 tasks every model ran. π0.5 is no longer listed there without explanation, so we keep its last published result.

    Real robots
  • updated Embodied First Exam

    GPT-6 Astra rescored from 33.33 to 33.77. The realtime mode was renamed and its default reset budget set to zero.

    Simulation
  • collected Embodied Analysis

    Now listed under collected but not ranked. It publishes only an Elo score per model, without naming its simulation tasks or publishing comparison counts or uncertainty.

    Simulation

  • added RoboColosseum open track

    • LingBot-VLA v2newReal robots

    Also covers Claude Opus 5.5, GalaxeaVLA G0.5, GPT-6 Astra, GR00T N1.7, Grok 4.7, MolmoAct2 and π0.5, which were already ranked.

    Real robots
  • added RoboColosseum fine-tune track

    • GalaxeaVLA G0.5Household · Precision
    • LingBot-VLA v2newReal robots

    Also covers GR00T N1.7, MolmoAct2 and π0.5, which were already ranked.

    Real robots
  • added 1 model to RoboDojo real

    • SLA-PreviewnewReal robots
    Real robots
  • added 1 model to RoboLab

    • FLUX 3 ActionnewOverall
    Simulation
  • added 1 model to PhAIL

    • FLUX 3 ActionnewOverall · Real robots
    Real robots
  • added 1 model to RoboColiseum

    • PyRUA-LeanSimulation
    Simulation
  • added 1 model to LIBERO-Pro2.0

    • WorldVLAnewSimulation
    Simulation
  • added 1 model to LIBERO-Plus

    • WorldVLAnewSimulation
    Simulation
  • added 4 models to EBench

    • Cosmos3-Edge-PolicySimulation
    • GigaBrain-0.7newNot yet ranked

    Also covers Fast-WAM and X-VLA, which were already ranked.

    Simulation
  • updated Embodied First Exam

    Protocol v2.2 rescored GPT-6 Astra, from 33.11 to 33.33.

    Simulation
  • updated RoboCasa365

    Agent submissions now need a paper rather than a rerun by the benchmark team.

    Simulation

  • added DataCenterBench

    • DeepSeek V4.1 FlashOverall · Real robots · Pick and place · Precision · Robustness
    • GLM 5.3 FlashOverall · Real robots · Precision
    • Grok 4.7newReal robots
    • Kimi K3newReal robots
    • Muse Wonder 1.0newOverall · Real robots

    Also covers Claude Opus 5.5, Gemini 4 Argon, GPT-6 Astra and GPT-6.1 Sol, which were already ranked.

    Real robots
  • added ArmnetBench

    • ACTHousehold
    • Diffusion PolicyOverall · Real robots · Pick and place · Household · Precision · Robustness
    • SmolVLAPrecision

    Also covers GR00T N1.7, MolmoAct2, π0 and π0.5, which were already ranked.

    Real robots
  • added VLA-Replica

    • ACTHousehold
    • SmolVLARobustness

    Also covers GR00T N1.7, MolmoAct2, X-VLA, π0 and π0.5, which were already ranked.

    Real robots
  • added EBench

    • InternVLA-A1.5newSimulation
    • InternW0-ΔSimulation · Robustness
    • PICA-WAMnewSimulation

    Also covers GPT-6 Astra, OpenWAM-α, π0 and π0.5, which were already ranked.

    Simulation
  • added 5 models to RoboDojo sim

    • EWAMnewSimulation
    • Meituan-Robotics-0newSimulation
    • HWM-0 PreviewnewNot yet ranked
    • Q0-PreviewnewNot yet ranked
    • WALL-OSS-1.0-litenewNot yet ranked
    Simulation
  • added 2 models to RoboTwin 2.0

    • EWAMnewSimulation
    • PICA-WAMnewSimulation
    Simulation
  • added 1 model to RoboCasa365

    • Meituan-Robotics-0newSimulation
    Simulation
  • added 1 model to PhAIL

    • GR00T N1.6Overall · Real robots · Pick and place · Household · Robustness
    Real robots
  • added 2 models to RoboColiseum

    • InternVLA-A1.5newSimulation
    • Muse Wonder 1.0newOverall
    Simulation
  • updated RoboColiseum

    Dream-1 was rerun. The new run replaces its earlier result.

    Simulation
  • updated Embodied First Exam

    Core test set rebuilt, from 449 to 456 tests.

    Simulation
  • updated PhAIL

    DROID track rescored over 286 tasks.

    Real robots

  • launched the index

    16 benchmarks, 68 models and 30 ranked overall.

This page is built from the same text as the methodology in Introducing the Robot Intelligence Index and is rebuilt with every index update, so the two always match.