LIBERO-Mem

LIBERO-Mem (AAAI 2026) is not the Spatial / Object / Goal / LIBERO-100 lifelong suites. FPT Software AI Center (with Arkansas, Stuttgart, CMU, Aalborg, and Liverpool) added 10 new tabletop tasks on the LIBERO / robosuite stack so a policy cannot succeed from the current frame. Cues live in object history: what moved, how many times, in what order, or behind an occlusion. The four official memory axes:

• Object Motion (OM) — recall the last pick / place on an aliased scene.
• Object Sequence (OS) — count repeated pick-and-place (3× / 5× / 7×).
• Object Relations (OR) — swap or rotate identical bowls across plates.
• Object Occlusion (OO) — remember which basket hid which object.

Each task ships 120 trajectories (100 train / 20 val), 200–700 frames, instance IDs, masks, and symbolic subgoals (→, ∨). Official scoring reports task success and subgoal completion. Embodied-SlotSSM is the authors' companion VLA — a model-card fact, not a WorldFoundry board row.

Paper: Rethinking Progression of Memory State in Robotic Manipulation. Project: libero-mem.github.io. Code: libero-mem/libero-mem.

WorldFoundry is official-results only. You run the upstream evaluator, then official-validation normalizes JSON/CSV into the scorecard. No in-tree simulator, no Embodied-SlotSSM weights. That is different from base LIBERO, which can also wire closed-loop execution.

Dimensions

• Object Motion (OM)

  KindAxis

  Recall the last pick or place when the current frame is aliased.

• Object Sequence (OS)

  KindAxis

  Count repeated pick-and-place (3× / 5× / 7×) that vision alone cannot disambiguate.

• Object Relations (OR)

  KindAxis

  Track temporal order when swapping or rotating identical bowls across plates.

• Object Occlusion (OO)

  KindAxis

  Remember which basket hid which object after it leaves the view.

Data composition

• Size

  10 new non-Markovian tabletop tasks on the LIBERO / robosuite stack (not Spatial / Object / Goal / 100). Each task: 120 trajectories (100 train / 20 val), 200–700 frames, object IDs, masks, and symbolic subgoals.

• Sources

  github.com

• Splits

  om · os · or · oo

• Modalities

  language · vision · action · state

• License

  upstream_terms

• Notes

  Official protocol also reports subgoal completion with Sequence (→) and Or (∨). WorldFoundry imports whatever the official export contains; it does not recompute subgoals.

Metrics

Metric

• success_rate

  DirectionHigher is better

  Full-instruction episode success imported from the official evaluator.

  PrimarySuccess Rate

• memory_success_rate

  DirectionHigher is better

  Success restricted to memory-critical instances when the export exposes that split.

• task_success

  DirectionHigher is better

  Per-task success among the 10 OM/OS/OR/OO tasks.

• action_accuracy

  DirectionHigher is better

  Offline action-prediction accuracy if the official dump includes it.

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Official LIBERO-Mem long-horizon suite from the paper. Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective (2025).

Model	T1–T10 average success
Naive SlotSSM	14.8%
π0 (h=1)	5.0%
SlotVLA (h=8)	5.0%
SlotVLA (h=1)	0%

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id libero-mem \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --output-dir tmp/libero-mem/official-validation \
  --json

Optional WORLDFOUNDRY_LIBERO_MEM_ROOT points at a local toolkit / repo root. Default import path when you omit --official-results-path is WORLDFOUNDRY_LIBERO_MEM_RESULTS_PATH. Manifest: worldfoundry/data/benchmarks/tasks/external/libero-mem.yaml. Outputs: scorecard.json (primary success_rate; also memory_success_rate when the export has it), raw_results.jsonl, and evaluation/summary.json.

← Benchmark Hub