Benchmarks/MolmoSpaces/Pick with Object Permanence

Pick with Object Permanence

MolmoSpacesMolmoSpaces-Commonsense
Robot
Franka FR3
Trials
1,000
Time limit
20 s
Step limit
300

Pick up the mug that the ball is hidden under (mugs fall at scene start, one covering the ball).

No public clip for this task
SuccessPass/fail

PickTask success evaluated on the correct mug, the one covering the ball. Mapping inferred from names.

Full criteria and notes
Full success check
PickTask success (lifted >= threshold, only robot contact) evaluated on the correct mug, the one covering the ball (MugBallPickTask). Mapping inferred from names. Episode counts as success if this holds at any step (oracle success, the ranking metric); at-end success is also reported for some tasks. Benchmark: molmospaces-commonsense-bench; max steps 300.
Scoring
1 per episode; task score = oracle successes / episodes x100.
Trials
~1,000 distinct episodes, each run once (median n=1000; varies by entry).

Task definition ↗ · All MolmoSpaces tasks