At a glance
- Sources
- 28 public leaderboards and result sets indexed, 23 of which feed the ranking.
- Data
- 164,783 rows pulled from each leaderboard's own data, each with its source and retrieval date.
- Models
- 102 entries matched by hand by name and version, counting 11 checkpoints trained per task on RoboChallenge as their own entries. Release dates are the first public paper, announcement, weights or listing.
- Rating
- Weighted Bradley–Terry fit to noise-aware head-to-head outcomes within each benchmark, on a 400-point chess-style scale centered at 1500.
- Uncertainty
- 300 re-runs resampling benchmarks, with related simulators drawn together, and score noise. Rank ranges cover 90% of re-runs.
- Checks
- Our fit to RoboArena's raw comparisons reproduces its ranking at 0.985.
What the index includes
Benchmarks. We included public robot leaderboards that publish a score for every model and were active in 2026. That covers benchmarks whose own teams run or verify each entry, such as RoboDojo, RoboChallenge, RoboArena, RoboLab, AgiBot's RoboColiseum, Positronic's PhAIL, Poke & Wiggle's Reality Check, the RoboColosseum team's real-world sites, Robocurve's RoboDojo-RC and DataCenterBench, Artificial Analysis's AA-Robotics, Armnet's ArmnetBench and UT Dallas's VLA-Replica, and the benchmarks most often reported in recent model papers, such as LIBERO, SimplerEnv and CALVIN, which are self-reported. A benchmark feeds the ranking only when some of its models also appear on another benchmark. We index 28 sources, and 23 of them feed the ranking. The benchmark table explains why the other 5 do not.
Coding agents. AA-Robotics tests only general coding agents, which write code to drive a real arm with no robot training. RoboDojo-RC tests the same kind of agent alongside robot models, and Robocurve's DataCenterBench tests agents on ten steps from data-center manuals. All three enter as real-robot benchmarks. RoboColosseum's open track also runs coding agents through Robocurve's harness, on the same real robots as robot models. AA-Robotics and DataCenterBench have lower benchmark weights because they score one to three attempts per task. RoboColiseum also tests coding agents next to robot models, in simulation.
Who gets ranked. By default the overall ranking includes a model only if it has at least one real-robot result and one simulation result. 36 models meet this threshold. The coverage menu also shows any two benchmarks (64 models) or every model (102). The simulation ranking requires two simulated benchmarks for a model to be ranked, but the real-robot ranking shows every model with a real result.
Ratings. Each benchmark is turned into head-to-head comparisons between the models it tested, and one rating per model is fitted to all of them on a chess-style Elo ranking where every model starts at 1500. The methodology walks through it step by step.
Task types
Beyond the overall rating, the index rates models on four types of task. We sorted each benchmark's own task and category scores into these types by hand, assigning each task to the type that best describes its main difficulty. Each type is then rated from those scores alone. A model's score in a task type is the plain average of its scores on that benchmark's tasks of that type. A model is ranked in a task type only if it meets the overall rule and has at least two benchmarks in that type.
| Task type | What it measures | Example tasks |
|---|---|---|
| Pick and place | Grasping objects and putting them in a target place or arrangement, usually in a few steps. | Stack bowls (RoboDojo, RoboChallenge, RoboDojo-RC). Sort books and place shoes on a rack (RoboChallenge). Place a toy in a marked rectangle (AA-Robotics). Move household objects between bins (PhAIL). Sort screws into bins (Poke & Wiggle). RoboLab's simple tasks and RoboColiseum's general manipulation track. |
| Household and long-horizon | Chores that chain several steps, objects or tools, where an early mistake can sink the whole task. | Make a vegetarian sandwich and wipe the table (RoboChallenge). Fold clothes and make toast (RoboDojo simulation). Pack and pour fruit (RoboDojo-RC). RoboCasa365's kitchen tasks and RoboLab's complex tasks. |
| Precision and dexterity | Tight tolerances, careful force or exact positioning. | Insert a key and fasten screws (RoboDojo simulation). Cap a pen and insert a charger (RoboDojo real). Plug in a network cable and scan a QR code (RoboChallenge). Pour a drink into a cup (AA-Robotics). Connect a DC jack and route a cable through closed hoops (Poke & Wiggle). |
| Robustness | Familiar tasks after the scene, camera, lighting, layout or wording changes. | Randomized scenes (RoboDojo simulation, RoboTwin 2.0). Perturbation suites (LIBERO-Plus, LIBERO-Pro2.0). Unseen multi-step tasks (RoboCasa365). Vague instructions (RoboLab). Objects placed outside the training range (Poke & Wiggle). RoboColiseum's robustness track. |
Benchmarks that report no breakdown count only toward the overall views, except where the whole benchmark tests robustness. EBench's held-out test and the perturbation scores of LIBERO-Plus and LIBERO-Pro2.0 also count toward robustness, as does VLA-Replica's out-of-distribution score. DataCenterBench counts toward precision only, because each of its tasks is one fine control or connector step. CALVIN's average chain length, from 0 to 5, is multiplied by 20 to put it on the same 0 to 100 scale as other scores. RoboArena counts toward pick and place because pick and place tasks dominate its task breakdown.
Indexed but not ranked
We index 28 sources, and 23 of them feed the ranking. The other 5 are collected with every update but do not feed it, for the reason given.
| Source | Type | Reason |
|---|---|---|
| PolaRiS | Real-to-sim | Publishes correlations only, no per-model scores. |
| BEHAVIOR Challenge 2025 | Simulation | Team entries share no models with any other board. |
| RoboChallenge Table30 V2, CVPR and ICRA tracks | Real robots | Only 2 entries cover all V2 tasks, and CVPR scores do not match the public runs. |
| LIBERO-PRO (original) | Simulation | Superseded by LIBERO-Pro2.0 from the same team. |
| Embodied Analysis, Physical Agent Capability Index | Simulation | Publishes only an Elo score per model. It does not name its simulation tasks or publish comparison counts or uncertainty. |
How ratings are computed
Each model gets one rating, built from head-to-head comparisons inside every benchmark it appears on. Every model starts at a score of 1500, and the gap between two ratings predicts how often one outscores the other.
1. Collect each benchmark's scores
We pulled every benchmark's public results as of November 17, 2026, and took one headline score per model from each. The table at the end of this section lists the score used for each benchmark. We matched the same model across benchmarks by hand, using name and version only. We leave out entries without a version or with an ambiguous name. A policy wrapped in a separate planner is not merged with the bare policy, and neither is a model trained on a benchmark's own data under a new name, such as π0.5-MolmoSpaces. Under this framework, a model fine-tuned separately for two benchmarks still counts as one model. A separate checkpoint trained for each task of one benchmark is different. We list those checkpoints as their own entry, marked per-task, so they do not stand in for the model they fine-tune. We discuss both choices in the limitations section.
Where a benchmark publishes success against time, we use its success at the longest time limit it reports, so a rating reflects whether a model finishes a task rather than how fast. Only PhAIL publishes this. Its DROID headline counts how much of each task is done within 240 s, and its success curves run to 480 s. We add each model's gain between the two on PhAIL's own curve to its headline score. Only GPT-6 Astra ran past 240 s, and it rises from 55.9 to 68.5, from fifth of seven to first. The other six had nearly stopped improving, gaining at most 1.7 points between 180 and 240 s. The noise on PhAIL's 480 s scores keeps the width of its published 240 s range.
RoboArena and RoboColosseum's open track publish ratings rather than success rates. We use each rating as the score and its published uncertainty as the noise, and count a RoboArena entry only once it has at least 20 evaluations. RoboColosseum rates each robot separately, so each robot is its own table, and its SO-101 table counts 0.71 times as much as the others because its ranking held up in only 59% of the benchmark's site splits, against about 85% on the other robots. When a benchmark re-runs a model, the new result replaces the old one. When a source cannot be reached, the index keeps the last data it retrieved.
2. Turn each benchmark into head-to-head matchups
Within a benchmark, every pair of models it tested is one matchup. Instead of a plain win or loss, each matchup records how likely it is that one model really scored higher, given how noisy that benchmark is.
qAB is the probability assigned to A outscoring B, sA and sB are their benchmark scores, and σA and σB are the assumed standard deviations of their score noise. Φ is the standard normal cumulative distribution function. Noise describes how much a score could move if the benchmark were run again. On RoboDojo's real robots, π0.5 scores 22.9 and X-VLA 7.6, each with a noise of 2.5 points. The gap is about four times the combined noise, so π0.5 counts as better with near certainty. GR00T N1.7 at 5.9 against π0 at 5.8 counts as 0.51, almost a tie.
Treating the two scores' noise as independent ignores that models tested on the same tasks tend to succeed and fail together. On a shared task set that usually overstates the noise in their difference, which makes matchups softer rather than more decisive, though other patterns of dependence can work the other way. For most benchmarks the noise is the sampling error plus a fixed 1 to 3 points, so a tiny gap between two precisely measured scores does not become a near-certain win. Where an entry has no published interval, RoboLab-120's noise is set to 2 points, and where MolmoSpaces gives no episode count we assume 1,000 episodes. Task-type scores use the same approach, with the sampling error computed from the attempts behind each average.
3. Weight each benchmark
In our index, each benchmark has a benchmark weight, which is how independent its evaluators are multiplied by how repeatable its scores are. We judge and score independence by rubric, and we take repeatability from published repeat runs where they exist and from attempts per task where they do not. Each matchup is scaled by the size of its benchmark to ensure a model’s result counts for roughly the same amount regardless of how many models are included in a benchmark's leaderboard. Benchmarks that publish several separate tables, such as MolmoSpaces' tasks or Poke & Wiggle's separate tables for robot models and agents, split their benchmark weight across those tables, and a model that appears in only one table gets only that table's share.
We set both factors with a fixed rubric. The evaluator factor is 1.0 when the benchmark's own team or an independent operator runs or verifies every entry. It is 0.9 when an independent party runs every entry but the evaluation has a known limit, such as volunteer labs running the comparisons, scoring without blinding, or an operator that fine-tunes the models it tests. It is 0.85 when a competing model's authors help run the benchmark, 0.8 when the operator runs only part of the evaluation, results mix operator runs and submissions, or the operator enters its own model, 0.75 when submitters run their own models and the operator also builds models, 0.7 for results posted in a benchmark's README and 0.4 for numbers self-reported in papers.
The repeatability factor comes from published reliability measurements for RoboChallenge (0.70, its repeat-run correlation) and RoboArena (0.90, its split-half agreement). For the other benchmarks it follows from how many attempts each runs per task and how they are scored, from 0.95 for the largest evaluations down to 0.55 for DataCenterBench, which scores three tries on each of ten tasks with LLM judges. The table below gives both factors for every benchmark. Because these judgments were subjective, we also tested robustness to several different weighting schemes, and found broad agreement between them. Section 9 reports those results in detail.
| Benchmark | Evaluator | Repeatability | Basis |
|---|---|---|---|
| RoboDojo (real) | 1.0 | 0.90 | Run by RoboDojo's team, 10 trials per task, blinded raters |
| RoboDojo-RC Tier 1 | 1.0 | 0.90 | Independent operator (Robocurve), 120 attempts per model |
| AA-Robotics | 1.0 | 0.60 | Independent operator (Artificial Analysis), one human-judged attempt per task |
| DataCenterBench Tier 1 | 1.0 | 0.55 | Independent operator on the RoboDojo-RC robots, three tries on each of ten tasks, judged by LLMs with a human tie-break |
| PhAIL | 1.0 | 0.85 | Operator-run, every episode published on video, about one attempt per DROID task |
| RoboArena | 0.9 | 0.90 | Double-blind comparisons run by volunteer labs; split-half agreement 0.90 |
| ArmnetBench | 0.9 | 0.85 | Third-party arm farm, about 360 rollouts per model, scored by the operator without blinding |
| VLA-Replica | 0.9 | 0.80 | Independent lab, 15 trials per task, reproduced at a second site |
| Poke & Wiggle Reality Check | 0.9 | 0.90 | Operator-run and rated by people, every episode on video; the operator sells evaluations and fine-tunes the robot models it tests; one sitting per model |
| RoboColosseum, open track | 0.85 | 0.85 | Blind comparisons on new tasks at many sites; MolmoAct2's authors help run it |
| RoboColosseum, fine-tune track | 0.85 | 0.80 | Operator-run, 25 trials per task; MolmoAct2's authors help run it |
| RoboChallenge | 0.8 | 0.70 | Operator's robots, with each submitter running its own model; repeat runs correlate at 0.70 |
| RoboDojo (simulation) | 1.0 | 0.95 | Run by RoboDojo's team, 150 episodes per task |
| RoboLab-120 | 1.0 | 0.95 | Run or cross-checked by the operator, 1,200 episodes |
| Embodied First Exam | 1.0 | 0.95 | Operator-run, 456 standard tests |
| LIBERO-Pro2.0 | 1.0 | 0.85 | One team runs every model's official checkpoint |
| RoboTwin 2.0 | 0.8 | 0.90 | Mix of operator runs and submissions, 100 trials per task |
| MolmoSpaces | 0.8 | 0.90 | Mix of operator runs and submissions, about 1,000 episodes per task |
| RoboCasa365 | 0.8 | 0.90 | Submitted, then verified by the operator, 50 rollouts per task |
| RoboColiseum | 0.8 | 0.85 | Run by AgiBot, which also enters its own model; episodes per task not published |
| EBench | 0.75 | 0.85 | Held-out test on the operator's server, with each submitter running its own model; the operator also builds models |
| LIBERO-Plus | 0.7 | 0.85 | Results posted in the benchmark's README |
| Paper-reported | 0.4 | 0.70 | Self-reported in papers |
Simulation results get two further adjustments, adopted on November 11 from our sim-to-real study. In the overall and task-type rankings every simulation benchmark counts half as much as it otherwise would, because simulators share errors and are not independent votes. Each simulator is also scaled by an agreement factor, from 0.88 to 1.17 today, that reflects how often it picks the same winner as other teams' real robots. The simulation ranking uses the agreement factors but not the halving. In the task-type rankings, paper-reported results keep a factor of 1. The sim-to-real agreement tables below list both.
4. Fit one rating per model
Our model uses two ratings to predict a matchup's outcome:
pAB is the predicted probability that A outscores B, and RA and RB are their fitted ratings.
We choose the set of ratings whose predictions best match all the weighted matchup outcomes at once. This is a weighted, regularized Bradley–Terry model fitted to the probabilistic matchups from step 2. Fitting Bradley–Terry ratings to results pooled across many benchmarks has precedent in machine learning, from comparing algorithms across data sets to ranking agents across Kaggle competitions.
Models that never met are placed through the models they share, with shared models acting as bridges between benchmarks. A small pull toward 1500 keeps a model with little evidence from landing at an extreme. The pull works like a normal prior centered on 1500 with a standard deviation of about 550 rating points. We fit the overall, simulation, real-robot and four task-type rankings separately, so a rating only compares models within one setting.
5. Read a rating
A gap between two ratings gives the predicted chance that the higher-rated model outscores the other.
| Rating gap | Predicted chance the higher-rated model outscores the other |
|---|---|
| 0 | 50% of the time |
| 100 | 64% |
| 200 | 76% |
| 400 | 91% |
The table above follows from the formula in step 4. We checked these predictions against unseen results, refitting the ratings 23 times, leaving out one benchmark each time, and predicting the order of every pair of models that benchmark tested with different scores, with 1,896 pairs in all. We find the fitted model's predictions strongly correlate with real outcomes. When the model gave the higher-rated model a 90 to 100% chance, that model outscored the other 97% of the time. In the 70 to 80% range it did so 83% of the time, meaning predictions in this band understate the winning model's relative strength. These checks only cover benchmarks like the ones in the index. Relative performance may vary on tasks that are out of distribution.
6. Measure uncertainty
We re-ran the whole fit 300 times. Each re-run draws benchmarks at random with replacement and nudges every score by its noise. A benchmark drawn more than once still counts once, so each re-run keeps about two thirds of the benchmarks and leaves out the rest. Related simulators are drawn together as one unit, the LIBERO family (LIBERO-Pro2.0, LIBERO-Plus and paper-reported results) and the two DROID-Franka simulators (RoboLab-120 and MolmoSpaces), so a blind spot they share cannot count twice. A model's rank range is where its rank falls in 90% of those re-runs, from the 5th to the 95th percentile, rounded down. In each re-run a model is ranked only against the models that re-run still includes. The leaderboard sorts by rating. Sorting by worst plausible rank instead uses the bottom of that range, which penalizes models with uncertain rankings. A model's range comes from the re-runs that include its benchmarks.
A model with only one benchmark in a leaderboard shows no range, since no re-run can test it against a second. When a re-run drops a benchmark that links two groups of models, the groups are placed relative to each other only by the pull toward 1500. In 2,000 separate draws, that happens in under 1% of re-runs overall, 2% in simulation and 3% of real-robot re-runs. Twelve real-robot benchmarks are in the final set. RoboDojo-RC and RoboColosseum's open track test coding agents and robot models on the same robots, and seven of RoboColosseum's models also appear on other benchmarks in the index. Real-robot rank ranges are therefore less reliable than the others.
7. Measure disagreement
For each benchmark in a view, we place the model by percentile among the models we matched there, from 100 for first to 0 for last. Where a benchmark has several tables in the view, its percentile is the average across them. We calculate spread as the sample standard deviation of those percentiles. The leaderboard shows it as ± N under each rating. It measures how much benchmarks disagree about a model, not the uncertainty in its rating, which the rank range covers. Dream-1 finishes first on RoboDojo-RC and in the 28th percentile on RoboColiseum, for a spread of 51. Claude Fable 5.2 finishes in the 91st percentile or higher on all four of its benchmarks, for a spread of 4. A model with one benchmark in the view has no spread. Because percentiles depend on which other models each benchmark tested, spread reflects differences in the level of competition as well as disagreement between benchmarks.
8. Decide who gets a rank
Every model's comparisons inform the full set of ratings, but only some models receive a rank. By default, the overall ranking requires at least one real-robot and one simulation result, the simulation ranking requires two simulated benchmarks, the real-robot ranking includes every model with a real result and the task types follow the overall rule while also requiring two benchmarks within the task type. We hold back entries whose underlying model or agent harness is undisclosed, or that were scored only after using several times the time other entries on the same benchmark were given, until their developers document them. AA-Robotics scores for GPT-6.2 Astra, GPT-6.2 Sol, Claude Fable 5.2, Gemini 4 Hydrogen and Gemini 4 Carbon, published on November 17 as a chart with no harness, effort setting or task scores, are held this way until Artificial Analysis publishes the runs. We also hold back entries that a benchmark itself marks as not following its rules, such as RoboDojo agent-track entries marked with an asterisk. An entry the benchmark also scores at the time limit the others had is not held back; step 1 explains which limit we use. Held entries stay in each benchmark's table but do not enter the fit. We make one exception. Claude Fable 5.2's RoboDojo simulation score comes from Anthropic's own runs in a harness it has not published, and RoboDojo has not listed it. We count it like RoboDojo's own entries and tag it as self-reported in the leaderboard.
9. Check robustness
We refit the overall ranking under alternatives to our input values for benchmark weights, the weight given to simulation, noise and the prior. The table compares every alternative with the published top five, Claude Fable 5.2, SLA-Preview, Dream-1, GPT-6 Astra and GPT-6.1 Sol.
| Alternative | First place | Top five |
|---|---|---|
| Equal benchmark weights, simulation at full weight and no agreement factors | Claude Fable 5.2 | 4 of 5, with InternW0-Δ replacing Dream-1 |
| Simulation at full weight, as before November 11 | Claude Fable 5.2 | 4 of 5, with InternW0-Δ replacing Dream-1 |
| Simulation at a third, or three quarters, of a real-robot result | Claude Fable 5.2 in both | 4 of 5 in both, with InternW0-Δ replacing GPT-6.1 Sol at a third and Dream-1 at three quarters |
| No agreement factors | Claude Fable 5.2 | Same five, same order |
| Evaluator factor only, or repeatability factor only | Claude Fable 5.2 | Same five in both, reordered with the repeatability factor only |
| Each benchmark weight scaled by a random factor between ½ and 2, drawn evenly on a log scale, 500 times | Claude Fable 5.2 in every draw | 4.3 of 5 kept on average, all five in 36% |
| Each benchmark weight scaled by a random factor between ¼ and 4, drawn evenly on a log scale, 500 times | Claude Fable 5.2 in 97%, SLA-Preview in 3%, Dream-1 in the rest | 3.8 of 5 kept on average |
| Each of the 23 benchmarks left out in turn | Claude Fable 5.2 in all 23 | Same five in 13 of 23. 4 of 5 in the other 10, with InternW0-Δ entering in 9 of them |
| Paper-reported results left out | Claude Fable 5.2 | Same five, same order |
| LIBERO family left out (LIBERO-Pro2.0, LIBERO-Plus and paper-reported) | Claude Fable 5.2 | Same five, same order |
| DROID-setup benchmarks left out (RoboArena and PhAIL) | Claude Fable 5.2 | Same five, same order. Cosmos3-Nano-Policy loses its only real-robot result |
| SO-101 benchmarks left out (ArmnetBench and VLA-Replica) | Claude Fable 5.2 | Same five, same order |
| RoboColosseum left out (both tracks) | Claude Fable 5.2 | Same five, same order |
| RoboDojo family left out (simulation, real robots and RoboDojo-RC) | Claude Fable 5.2 | 3 of 5. SLA-Preview loses its only real-robot result and Dream-1 loses RoboDojo-RC, and Cosmos3-Nano-Policy and Claude Opus 5.5 enter |
| All noise halved, or all noise doubled | Claude Fable 5.2 in both | Same five, same order, when halved. 3 of 5 when doubled, with InternW0-Δ and MWA-Phoenix replacing GPT-6 Astra and GPT-6.1 Sol |
| Pull toward 1500 five times weaker, or four times stronger | Claude Fable 5.2 in both | 4 of 5 in both, with InternW0-Δ replacing GPT-6 Astra when weaker and Dream-1 when stronger |
Claude Fable 5.2's first place holds by a wide margin in every alternative, including leaving out RoboDojo-RC, Poke & Wiggle or its self-reported RoboDojo simulation score, and leaving out the whole RoboDojo family. Places two to nine depend more on these choices. SLA-Preview's second place rests on RoboDojo's real-robot track, its only real-robot result, which it leads by 3.4 points, and it stays second or third in 90% of the random reweightings between ½ and 2. Dream-1's place rests on the higher weight for real-robot results. It leads RoboDojo-RC, its only real-robot result, and finishes 24th of 30 on RoboColiseum, its only simulation result, so it leaves the top five when simulation counts three quarters as much as a real result or more. RoboDojo's real-robot track gives SLA-Preview, InternW0-Δ, MWA-Phoenix and OpenWAM-α their only real-robot result, and RoboColiseum gives Dream-1 its only simulation result. PhAIL gives FLUX 3 Action and Cosmos3-Nano-Policy their only real-robot result.
Benchmark weights and noise
| Benchmark | Score used | Benchmark weight | Noise per score | Weight in overall ranking |
|---|---|---|---|---|
| RoboDojo (real) | Overall progress score | 0.90 | 2.5 points | 0.90 |
| RoboDojo-RC Tier 1 | Mean progress over 120 attempts | 0.90 | From its spread across attempts, about 3 points | 0.90 |
| AA-Robotics | Mean task score over 8 tasks | 0.60 | About 14 points, one attempt per task | 0.60 |
| DataCenterBench Tier 1 | Mean score over 10 tasks, three tries each | 0.55 | Its published 95% range | 0.55 |
| ArmnetBench | Strict success over 12 tasks | 0.77 | Sampling error over about 360 rollouts, plus 2 | 0.77 |
| VLA-Replica | In-distribution success over 10 tasks | 0.72 | Sampling error over 150 trials, plus 2 | 0.72 |
| RoboChallenge | Progress over 30 tasks | 0.56 | About 3.7 points, from its repeat runs | 0.56 |
| RoboArena | Head-to-head rating | 0.81 | Its published uncertainty | 0.81 |
| RoboColosseum, open track | Rating on each robot, weight split | 0.72 | Its published uncertainty | 0.72 |
| RoboColosseum, fine-tune track | Progress over 20 tasks, by robot | 0.68 | Sampling error over 25 trials per task, plus 2 | 0.68 |
| PhAIL | Reliability on 286 DROID tasks, at 480 s, and on pick and place, weight split | 0.85 | Its published 95% range | 0.85 |
| Poke & Wiggle Reality Check | Success over 10 tasks, robot models and agents in separate tables, weight split | 0.81 | Sampling error over 5,400 or 900 episodes, plus 2 | 0.81 |
| RoboDojo (simulation) | Overall progress score | 0.95 | Sampling error over 6,300 episodes, plus 1 point | 0.42 |
| RoboTwin 2.0 | Average of clean and randomized | 0.72 | 3 points | 0.42 |
| RoboLab-120 | Success rate | 0.95 | Its published 95% interval | 0.51 |
| MolmoSpaces | Success per task, weight split | 0.72 | Sampling error over about 1,000 episodes, plus 3 | 0.36 |
| RoboCasa365 | Overall success rate | 0.72 | Sampling error over about 900 episodes, plus 3 | 0.36 |
| RoboColiseum | Total score over 38 tasks | 0.68 | Sampling error assuming about 20 episodes per task, plus 2 | 0.34 |
| EBench | Success on the held-out test | 0.64 | Sampling error over 794 instances, plus 2 | 0.35 |
| LIBERO-Pro2.0 | Success under perturbations | 0.85 | 2 points | 0.39 |
| LIBERO-Plus | Success under perturbations | 0.60 | 3 points | 0.30 |
| Embodied First Exam | Share of 456 standard tests solved | 0.95 | Sampling error over 456 tests, plus 1 | 0.51 |
| Paper-reported | LIBERO, LIBERO-Long, SimplerEnv WidowX and Google Robot, CALVIN and RoboCasa, weight split six ways | 0.28 | 4 points | 0.12–0.14 |
Benchmark weight is the evaluator factor multiplied by the repeatability factor, both listed in step 3. The last column is the benchmark's weight in the overall ranking, after the halving and agreement factor for simulators. Giving every benchmark equal weight, with simulation at full weight and no agreement factors, keeps four of the top five, with InternW0-Δ in place of Dream-1, and rank correlation 0.97 overall and 0.90 or higher in every other leaderboard. Section 9 reports wider changes to the weights.
Sim-to-real agreement
Our sim-to-real study found that simulators from different teams often share the same mistakes, and that a simulator picks the same winner as another team's real robot about as often as two real-robot benchmarks from different teams agree. We publish these rates with every update, recomputed on the index's current data. A benchmark decides a matchup when its score gap would survive a rerun at least 90% of the time under the noise in step 2, and two benchmarks agree when both decide a matchup and pick the same winner. We never compare two benchmarks run by the same team, and we split paper-reported results by suite.
| Comparison | Same winner | Matchups |
|---|---|---|
| Simulators against other teams' real robots | 80% | 229 |
| Real-robot benchmarks from different teams | 82% | 50 |
| Simulators from different teams | 86% | 374 |
| Simulator | Agrees with other teams' real robots | Matchups | Agreement factor |
|---|---|---|---|
| RoboDojo (simulation) | 70% | 40 | 0.88 |
| RoboColiseum | 81% | 32 | 1.01 |
| RoboTwin 2.0 | 97% | 31 | 1.17 |
| LIBERO-Pro2.0 | 70% | 23 | 0.92 |
| Embodied First Exam | 89% | 19 | 1.07 |
| EBench | 100% | 14 | 1.11 |
| RoboLab-120 | 93% | 14 | 1.07 |
| LIBERO, paper-reported | 73% | 11 | 0.97 |
| RoboCasa, paper-reported | 55% | 11 | 0.89 |
| CALVIN, paper-reported | 70% | 10 | 0.96 |
| SimplerEnv Google Robot, paper-reported | 62% | 8 | 1 (too few) |
| SimplerEnv WidowX, paper-reported | 62% | 8 | 1 (too few) |
| MolmoSpaces | 100% | 5 | 1 (too few) |
| RoboCasa365 | 100% | 2 | 1 (too few) |
| LIBERO-Plus | 100% | 1 | 1 (too few) |
| Model families in the matchup | Simulator agrees with real robot | Matchups |
|---|---|---|
| VLA against VLA | 74% | 138 |
| LLM against LLM | 86% | 28 |
| VLA against Other | 95% | 22 |
| WAM against VLA | 100% | 21 |
| VLA against LLM | 100% | 15 |
| WAM against LLM | 0% | 4 |
| WAM against WAM | 0% | 1 |
The agreement factor pulls each simulator's agreement toward the pooled rate, as if it had 71 more matchups at that rate, then rescales it so a simulator at the pooled rate gets 1 and one at chance gets 0, within limits of 0.25 and 1.5. Simulators with fewer than 10 decided matchups keep a factor of 1. The differences between simulators are still within noise, so the factors stay close to 1 for now and will spread as shared real-robot results accumulate. Listing per-task checkpoints separately raised overall agreement from 75% to 79% at the time.
Cost and time
The leaderboard, benchmark and model pages show what one task attempt costs and how long it takes. Few benchmarks publish either, so most of these figures are Playground estimates, tagged est. in the tables. They cover the compute to run a model, not the robot, its operators or the evaluation, and they play no part in the rating.
Time per attempt
| Source | Results |
|---|---|
| Published by the benchmark, or derived from its own run data | 81 of 273 |
| Estimated from the benchmark’s time limit as limit × (1 − success ÷ 2), so a model that always succeeds is assumed to finish in half the limit and one that never does to use all of it. Benchmarks that publish ratings rather than success rates use 50%. | 176 of 273 |
| The median measured time on the same benchmark, where it publishes no time limit | 16 of 273 |
Time limits on simulated benchmarks are in simulated seconds, so simulated times describe the length of an episode rather than the computing time to run it.
Cost per attempt
| Source | Results |
|---|---|
| Published by the benchmark | 32 of 273 |
| A benchmark’s published total spread over its rated trials, or, where a model has none, another model’s published cost on the same benchmark scaled by API price | 6 of 273 |
| For coding agents without a published cost, the agent’s own median published cost per attempt on other benchmarks, or the median across agents when it has none | 19 of 273 |
| For robot models served through a paid API, the provider’s price per robot-hour × time | 2 of 273 |
| For other robot models, one H100 at $3.29 per hour (Lambda on-demand, Oct 2026) × time | 214 of 273 |
A coding agent’s cost is what its API calls cost, while a robot model’s is the price of renting the hardware it runs on for the length of the attempt. The two are different kinds of cost, so the rating-against-cost charts compare agents and robot models only roughly.
In the leaderboard
A model’s cost and time in a leaderboard view are the median across the benchmarks in that view, which is the average of the two middle values when there is an even number. They are tagged est. when any benchmark behind them is estimated. Model pages give the figure for each benchmark and the same median across all of the model’s benchmarks, and each benchmark page lists the figure for every model it tested.
Updates, corrections and disclosures
Updates
We check every source two to three times a day and publish an update whenever a result changes. Each update refits every ranking, rank range and robustness check, and recomputes the sim-to-real agreement factors, so a simulator's weight can shift slightly from one update to the next. The changelog records what each update added, and we keep a copy of the leaderboard from every update.
Corrections
When a benchmark corrects or re-runs a result, the new result replaces the old one in the next update, and the changelog notes the change. To report an error, email contact@withplayground.ai with the model, the benchmark and a link to the source. We check every report against the benchmark's published data. A confirmed error is fixed in the next update and noted in the changelog.
Independence and disclosures
Evaluation runs are conducted independently of Playground. Every score in the index comes from a benchmark's own published results or, where marked, a developer's self-reported results. Playground also offers evaluations to model developers, and we disclose all of Playground's relationships with model providers.
Limitations
Why combine benchmarks
No single robot benchmark tests more than a fraction of today's policies, and each covers a different mix of robots, tasks and evaluation protocols. Combining them gives the broadest comparison currently available. A model's rating reflects how it performs against many others across many settings, rather than on one task suite chosen by one team. The method is built to show where that comparison is strong and where it is thin, through rank ranges, spread and coverage thresholds.
Benchmark-specific checkpoints
Some policies are fine-tuned for a specific benchmark before evaluation. A rating therefore describes a model as its developers or each benchmark's team adapted and ran it. That matches how these policies are used in practice, but it means a rating cannot separate a model's design from the training it received for each benchmark. The spread column makes cross-benchmark disagreement visible, and helps surface cases where models were optimized to max out a specific eval suite. The exception is a separate checkpoint trained for each task of one benchmark, as some developers entered on RoboChallenge. We list those as their own entries, so CogACT, OpenVLA-OFT and RDT, whose only real-robot result was a per-task entry, are not ranked overall until another real-robot benchmark tests them.
Hardware and harness differences
Benchmarks run on different robots and give coding agents different tools. Real-robot results come from setups as different as a one-arm Hiwonder NexArm on AA-Robotics, the two-arm I2RT YAM on RoboDojo-RC and DataCenterBench, low-cost SO-101 arms on ArmnetBench and VLA-Replica, two-arm ARX X5 and Piper robots on RoboDojo, Franka arms on RoboArena and PhAIL, two-arm Franka FR3 stations on Poke & Wiggle, and Franka, two-arm YAM, SO-101 and Unitree G1 robots at everyday sites such as offices and parks on RoboColosseum. Coding agents also run inside different harnesses. Embodied First Exam pairs each model with an agent harness, such as Codex for GPT-6 Astra, Antigravity for Gemini 4 Argon and Claude Code for Claude Opus 5.5. RoboDojo-RC instead runs its LLMs with a shared system prompt that includes rig calibration, grasping advice and a 40-call limit. RoboColosseum runs them through Robocurve's harness at medium reasoning effort, with up to 30 model calls per trial. Poke & Wiggle has each agent write robot code on its own skill library from ten demonstrations, with up to 30 simulated and 10 real practice trials before the rated ones. A single rating folds these differences together, so it describes a model across the mix of robots and harnesses its benchmarks used, not on any one setup. Because every comparison happens inside one benchmark, a setup that is harder for every model does not change the ranking. Differences matter only when a model suits one setup more than another. RoboDojo measures this directly by scoring the same 13 policies on three arms, each with its own six tasks. The same four policies lead on every arm, though a different one ranks first on each, and rank correlation between arms runs from 0.76 to 0.82.
Overlap between benchmarks
Ratings are comparable only through models that appear on more than one benchmark. Every benchmark in the index shares models with at least one other, and the fit uses each shared model as a bridge. Some connections rest on very few models. Coding agents meet robot models on five benchmarks. Among real-robot benchmarks, RoboColosseum's open track puts three coding agents and five robot models on the same robots, and only GR00T N1.7 and MolmoAct2 link RoboDojo-RC to the rest. This means a shift in how one such model was prepared can move a whole group of models, which rank ranges only partially capture.
Which models each benchmark tests
Developers choose where to submit, so newer models tend to appear on fewer benchmarks. Some of the newest VLAs, including Awomo-0.5I, Axiom-0 and Xiaomi-Robotics-1, have only simulation results so far and will join the default overall ranking once a real-robot benchmark tests them. Until then they appear when the coverage menu includes every model, and the worst-plausible-rank sort favors models whose position holds up across benchmarks.
Benchmark design
Benchmarks emphasize different skills. RoboDojo-RC chose its tasks after trying GPT-6 Astra, and half of RoboColiseum's total score comes from instruction following and spatial reasoning, where coding agents do well. Benchmark weights, set by independence and repeatability, limit how much any one design can move the ratings. Giving every benchmark equal weight produces nearly the same order, with rank correlation 0.97 overall and 0.90 or higher in every task-type leaderboard, and the task-type leaderboards show how models compare on specific kinds of manipulation.
Operators who also compete
AgiBot runs RoboColiseum and also trained GO-3 Nano Preview, second in our simulation ranking. Every RoboColiseum entry is scored by the same automated evaluation. We give the benchmark a lower evaluator factor (0.8) to account for the overlap. MolmoAct2's authors help run RoboColosseum, where MolmoAct2 competes. Its open-track evaluators do not know which models they are comparing, and we give it a lower evaluator factor (0.85) too.
Judgment calls
The weights, noise levels, task sorting and model matching are judgment calls. We've documented our process for assigning these values in the methodology. We further test them two ways. First, applied to RoboArena’s raw comparisons alone, our fitting method recovers RoboArena’s own ranking (rank correlation 0.985). Second, section 9 refits the combined ranking under alternative weights, noise levels and benchmark sets, and finds little change in relative rank. The half weight for simulation is a judgment call too. At a third, and at three quarters, Claude Fable 5.2 stays first and four of the top five stay.
What comes next
As more benchmarks publish results and more models appear on several of them, the overlap behind these ratings will grow and rank ranges should narrow. We will update the index as new results appear, and publish its sim-to-real agreement with each update.
Changelog
The changelog tracks changes in benchmark methodology, new benchmarks added to the index, and new models added by existing benchmarks. We preserve historical copies of the leaderboard with each update. Please contact us directly for historical leaderboard data.
- Real robots
updated AA-Robotics
Artificial Analysis charted new scores for GPT-6.2 Astra (79), GPT-6.2 Sol (78), Claude Fable 5.2 (75), Gemini 4 Hydrogen (58) and Gemini 4 Carbon (43) without the harness, effort setting or task scores it published for its first runs. We briefly counted them, then held them until Artificial Analysis publishes the runs. Ratings are as on November 16.
- Simulation
added 1 model to RoboDojo sim
- Era0-PreviewnewNot yet ranked
- Real robots
added 2 models to Poke & Wiggle Reality Check
- Mistral Large 4newReal robots
Also covers Gemini 4 Carbon, which was already ranked.
- Simulation
added 1 model to RoboTwin 2.0
- SLA-PreviewOverall
- Simulation
added 1 model to Embodied First Exam
- GPT-6.1 SolSimulation · Robustness
- Simulation
updated RoboDojo sim
Meituan-Robotics-0 was re-evaluated by its team, from 14.95 to 21.68. The new run replaces its earlier result.
- Real robots
updated PhAIL
Some DROID episodes were re-graded. Scores moved by at most 0.2 points. The index now counts DROID tasks at 480 s, the longest limit PhAIL publishes, rather than its 240 s headline, so the rating measures whether a model finishes a task, not how fast. Only GPT-6 Astra ran past 240 s; its DROID value rises from 55.9 to 68.5.
- Real robots
added Poke & Wiggle Reality Check
- Claude Fable 5.2Robustness
- DiT-FlownewReal robots
- RhonewReal robots
Also covers Claude Opus 5.5, Gemini 4 Argon, GPT-6 Astra, GR00T N1.7, MolmoAct2 and π0.5, which were already ranked.
- Simulation
updated RoboColiseum
FF Omnimind reran general manipulation, raising its total score from 0.902 to 0.946.
- Real robots
added 1 model to PhAIL
Also covers GPT-6 Astra, which was already ranked.
- Simulation
added 1 model to Embodied First Exam
- Claude Fable 5.2Simulation · Pick and place · Household · Precision
- Simulation
updated Embodied First Exam
Codex + GPT-6 Astra solves 157 of 456 standard tests after reruns, up from 153. The exam keeps each agent's latest finished attempt per test.
- Real robots
updated PhAIL
DROID adds a November sitting on 109 tasks run beside four board models. GPT-6 Astra joins at 55.9. π0.5 and GR00T were rerun after serving fixes and now count only their new runs, π0.5 rising from 12.0 to 62.5 and GR00T from 36.0 to 62.3.
updated the methodology
Adopted the four changes from our sim-to-real study. Simulation results now count half as much as real-robot results in the overall and task-type rankings, and each simulator is also weighted by how often it agrees with other teams' real robots. Rank ranges resample the LIBERO family and the two DROID-Franka simulators together. Checkpoints trained separately for each RoboChallenge task are listed as their own entries, so CogACT, OpenVLA-OFT and RDT leave the overall ranking until another real-robot benchmark tests them. The index's sim-to-real agreement is now published with every update.
- Simulation
added 1 model to RoboDojo sim
- Claude Fable 5.2newOverall
- Real robots
added 1 model to RoboDojo-RC
- Claude Fable 5.2newOverall · Real robots
- Simulation
added 1 model to RoboLab
- Praxis-1newNot yet ranked
- Simulation
added 1 model to LIBERO-Pro2.0
- CloudRobo-WM 1.0newNot yet ranked
- Simulation
updated LIBERO-Pro2.0
CloudRobo-WM 1.0 reports success rates from its own evaluation run. The benchmark's other entries come from official checkpoints run by its authors.
- Real robots
updated RoboDojo-RC
Claude Fable 5.2 comes from Robocurve's own runs with the main board's tasks and judge. Robocurve has not yet put it on its main leaderboard.
- Simulation
updated RoboDojo sim
Claude Fable 5.2's 64.9 is self-reported by Anthropic from its own harness. RoboDojo has not listed it, and we count it as a RoboDojo simulation result.
- Real robots
added 1 model to RoboDojo-RC
- Gemini 4 CarbonnewReal robots
- Real robots
updated RoboDojo-RC
Gemini 4 Carbon, an early-access Google model, now appears on Robocurve's main leaderboard.
- Simulation
updated RoboColiseum
FF Omnimind reran its robustness dimension. The new run replaces its earlier result. Two new entrants, MXI SimWorld v0 Preview and HexaShell, are listed but not yet counted until their developers document the model and harness they run, since each used several times the evaluation time of other entries.
- Simulation
added 1 model to EBench
- Motubrain 2.0newNot yet ranked
- Simulation
updated RoboColiseum
FF Omnimind reran two of its four dimensions. The new run replaces its earlier result.
- Simulation
added 3 models to RoboColiseum
- GigaBrain-0.7Simulation
- LingBot-VLA v2Overall · Pick and place
- FF OmnimindnewNot yet ranked
- Simulation
added 1 model to Embodied First Exam
- Kimi K3.1 PronewNot yet ranked
- Simulation
updated Embodied First Exam
The core view was removed, so we now use the standard view over the same 456 tests. Re-runs replaced several earlier results.
- Simulation
updated RoboLab
New submissions are paused while the team revises its submission process.
- Real robots
updated PhAIL
DROID tasks rescored over the 278 tasks every model ran. π0.5 is no longer listed there without explanation, so we keep its last published result.
- Simulation
updated Embodied First Exam
GPT-6 Astra rescored from 33.33 to 33.77. The realtime mode was renamed and its default reset budget set to zero.
- Simulation
collected Embodied Analysis
Now listed under collected but not ranked. It publishes only an Elo score per model, without naming its simulation tasks or publishing comparison counts or uncertainty.
- Real robots
added RoboColosseum open track
- LingBot-VLA v2newReal robots
Also covers Claude Opus 5.5, GalaxeaVLA G0.5, GPT-6 Astra, GR00T N1.7, Grok 4.7, MolmoAct2 and π0.5, which were already ranked.
- Real robots
added RoboColosseum fine-tune track
- GalaxeaVLA G0.5Household · Precision
- LingBot-VLA v2newReal robots
Also covers GR00T N1.7, MolmoAct2 and π0.5, which were already ranked.
- Real robots
added 1 model to RoboDojo real
- SLA-PreviewnewReal robots
- Simulation
added 1 model to RoboLab
- FLUX 3 ActionnewOverall
- Real robots
added 1 model to PhAIL
- FLUX 3 ActionnewOverall · Real robots
- Simulation
added 1 model to RoboColiseum
- PyRUA-LeanSimulation
- Simulation
added 1 model to LIBERO-Pro2.0
- WorldVLAnewSimulation
- Simulation
added 1 model to LIBERO-Plus
- WorldVLAnewSimulation
- Simulation
added 4 models to EBench
- Cosmos3-Edge-PolicySimulation
- GigaBrain-0.7newNot yet ranked
Also covers Fast-WAM and X-VLA, which were already ranked.
- Simulation
updated Embodied First Exam
Protocol v2.2 rescored GPT-6 Astra, from 33.11 to 33.33.
- Simulation
updated RoboCasa365
Agent submissions now need a paper rather than a rerun by the benchmark team.
- Real robots
added DataCenterBench
- DeepSeek V4.1 FlashOverall · Real robots · Pick and place · Precision · Robustness
- GLM 5.3 FlashOverall · Real robots · Precision
- Grok 4.7newReal robots
- Kimi K3newReal robots
- Muse Wonder 1.0newOverall · Real robots
Also covers Claude Opus 5.5, Gemini 4 Argon, GPT-6 Astra and GPT-6.1 Sol, which were already ranked.
- Real robots
added ArmnetBench
- ACTHousehold
- Diffusion PolicyOverall · Real robots · Pick and place · Household · Precision · Robustness
- SmolVLAPrecision
Also covers GR00T N1.7, MolmoAct2, π0 and π0.5, which were already ranked.
- Real robots
added VLA-Replica
- ACTHousehold
- SmolVLARobustness
Also covers GR00T N1.7, MolmoAct2, X-VLA, π0 and π0.5, which were already ranked.
- Simulation
added EBench
- InternVLA-A1.5newSimulation
- InternW0-ΔSimulation · Robustness
- PICA-WAMnewSimulation
Also covers GPT-6 Astra, OpenWAM-α, π0 and π0.5, which were already ranked.
- Simulation
added 5 models to RoboDojo sim
- EWAMnewSimulation
- Meituan-Robotics-0newSimulation
- HWM-0 PreviewnewNot yet ranked
- Q0-PreviewnewNot yet ranked
- WALL-OSS-1.0-litenewNot yet ranked
- Simulation
added 2 models to RoboTwin 2.0
- EWAMnewSimulation
- PICA-WAMnewSimulation
- Simulation
added 1 model to RoboCasa365
- Meituan-Robotics-0newSimulation
- Real robots
added 1 model to PhAIL
- GR00T N1.6Overall · Real robots · Pick and place · Household · Robustness
- Simulation
added 2 models to RoboColiseum
- InternVLA-A1.5newSimulation
- Muse Wonder 1.0newOverall
- Simulation
updated RoboColiseum
Dream-1 was rerun. The new run replaces its earlier result.
- Simulation
updated Embodied First Exam
Core test set rebuilt, from 449 to 456 tests.
- Real robots
updated PhAIL
DROID track rescored over 286 tasks.
launched the index
16 benchmarks, 68 models and 30 ranked overall.
This page is built from the same text as the methodology in Introducing the Robot Intelligence Index and is rebuilt with every index update, so the two always match.