MolmoSpaces-Bench

Last updated: Oct 7, 2026

On this page

Dimensions Data composition Metrics Leaderboard How to run

MolmoSpaces-Bench is Allen Institute for AI's simulation eval for MolmoBot / MolmoSpaces spatial-language manipulation. The paper's training dump (MolmoBot-Data) is 1.7 million expert trajectories in an ecosystem cited as 232k environments and 48k manipulable objects — that is not the bench. The bench is a 1,000-episode ladder: Pick MSProc / Classic / photorealistic Pick / Pick Random-Cam (20 s), then Pick&Place, PnP Next-To (5 cm), PnP Color (40 s). Official tables split pick-stage success from final / oracle success. A public board lives at molmospaces.allen.ai/leaderboard; this hub has not ingested its rows.

Paper: MolmoBot. Project: allenai.github.io/MolmoBot. Code: allenai/MolmoBot. WorldFoundry's contract split name is pick_and_place.

WorldFoundry imports official rollout exports. It does not run MolmoSpaces, generate the 1.7M set, or copy the Ai2 board into this page.

Official paper teaser: MolmoBot zero-shot manipulation across simulated MolmoSpaces.

Dimensions

Pick variants
KindTask
Pick MSProc, Pick Classic, photorealistic Pick, and Pick Random-Cam. Official sim episodes are 1,000-long; pick budget is 20 seconds.
Pick-and-place variants
KindTask
Pick&Place (into a receptacle), PnP Next-To (within 5 cm), PnP Color. 40-second budget. WorldFoundry's contract split name is pick_and_place.
Pick-stage vs final success
KindAxis
Official tables report pick-stage success separately from full-task / oracle final success.

Data composition

Size
Ai2 eval suite for MolmoBot / MolmoSpaces. Training data (MolmoBot-Data) is 1.7 million expert trajectories in an ecosystem the paper cites as 232k environments and 48k manipulable objects. The bench itself is the 1,000-episode simulation ladder plus a real-robot tabletop protocol — not the 1.7M set. Official board: molmospaces.allen.ai/leaderboard.
Sources
github.com
Splits
pick_and_place
Modalities
language · vision · action · state
License
upstream_terms
Notes
An official leaderboard exists. This hub has not ingested its rows. WorldFoundry only normalizes a caller-provided export.

Metrics

Metric

success_rate
DirectionHigher is better
Full-task / final episode success on the imported split.
PrimarySuccess Rate
pick_success_rate
DirectionHigher is better
Pick-stage success, reported separately from place / final success.
task_success
DirectionHigher is better
Per-task success aggregate (Pick vs Pick&Place variants).
reward
DirectionHigher is better
Mean episode reward / return when the export includes it.

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. molmospaces.allen.ai.

molmospaces.allen.ai · LiveOpen live board

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id molmospaces \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --output-dir tmp/molmospaces/official-validation \
  --json

Optional WORLDFOUNDRY_MOLMOSPACES_ROOT points at a local runtime/repo. Default import path: WORLDFOUNDRY_MOLMOSPACES_RESULTS_PATH. Manifest: worldfoundry/data/benchmarks/tasks/external/molmospaces.yaml. Outputs: scorecard.json (primary success_rate; also pick_success_rate), raw_results.jsonl, evaluation/summary.json.

← Benchmark Hub