WorldFoundry
English中文

RoboMME

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

RoboMME (University of Michigan with Stanford and Figure AI; ICML 2026) is a standardized bench for memory-augmented VLAs. 16 long-horizon tasks sit in four cognitive suites — Counting (temporal: exact event counts), Permanence (spatial: occlusion and swaps), Reference (object identity from an earlier cue), Imitation (procedural: copy a demonstrated trajectory) — four tasks each. The release includes 1,600 demonstrations and 770k training timesteps, plus 14 memory-augmented variants on a π0.5 backbone. The paper reports per-suite success, not a single pick-and-place bit.

Paper: RoboMME. Project: robomme.github.io. Code: RoboMME/robomme_policy_learning.

WorldFoundry imports official rollout exports and normalizes overall plus per-suite rates. It does not run the RoboMME simulator or those π0.5 variants. Name the suite (counting / permanence / reference / imitation) on the export.

Official paper teaser: memory-augmented manipulation suites for generalist robot policies.
Official paper teaser: memory-augmented manipulation suites for generalist robot policies.

Dimensions

  • Counting
    KindSuite

    Temporal memory: accumulate events and stop after an exact count (four tasks).

  • Permanence
    KindSuite

    Spatial memory: track objects through occlusion and swaps (four tasks).

  • Reference
    KindSuite

    Object memory: resolve identity from an earlier visual or language cue (four tasks).

  • Imitation
    KindSuite

    Procedural memory: reproduce a demonstrated trajectory (four tasks).

Data composition

  • Size

    ICML 2026 memory-augmented manipulation bench from Michigan, Stanford, and Figure AI. 16 long-horizon tasks (4 suites × 4), 1,600 demonstrations, 770k training timesteps. The paper also releases 14 memory-augmented VLA variants on a π0.5 backbone. Built for history-dependent VLA evaluation, not a single binary pick-and-place bit.

  • Sources

    github.com

  • Splits

    counting · imitation · permanence · reference

  • Modalities

    language · vision · action · state

  • License

    upstream_terms

  • Notes

    WorldFoundry normalizes per-suite success from an official export. It does not run the RoboMME simulator or the π0.5 variants.

Metrics

Metric

  • success_rate
    DirectionHigher is better

    Overall episode success across imported suites.

    PrimarySuccess Rate

  • counting_success_rate
    DirectionHigher is better

    Success on the Counting suite (temporal memory).

  • permanence_success_rate
    DirectionHigher is better

    Success on the Permanence suite (spatial memory).

  • reference_success_rate
    DirectionHigher is better

    Success on the Reference suite (object memory).

  • imitation_success_rate
    DirectionHigher is better

    Success on the Imitation suite (procedural memory).

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Paper AVG column. Oracle rows use privileged memory. RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies (2026).

ModelAverage success (16 tasks)
Human90.50
MME-VLA GroundSG Oracleprivileged84.08
MME-VLA SimpleSG Oracleprivileged49.58
SimpleSG QwenVL29.00
Gemini (SimpleSG Oracle setting)23.25

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id robomme \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --output-dir tmp/robomme/official-validation \
  --json

Optional WORLDFOUNDRY_ROBOMME_ROOT points at a local runtime/repo. Default import path: WORLDFOUNDRY_ROBOMME_RESULTS_PATH. Manifest: worldfoundry/data/benchmarks/tasks/external/robomme.yaml. Outputs: scorecard.json (primary success_rate; also counting_success_rate, permanence_success_rate, reference_success_rate, imitation_success_rate), raw_results.jsonl, evaluation/summary.json.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets