LIBERO-Pro

Last updated: Oct 7, 2026

On this page

Dimensions Data composition Metrics Leaderboard How to run

LIBERO-PRO (Huazhong University of Science and Technology, with Harvard, MIT, WUT, and Lehigh) is not a new lifelong suite and not LIBERO-Plus's 10,030-instance factor grid. It is a plug-and-play overlay on the original LIBERO tasks. The claim is that stock LIBERO train/eval are almost the same scene, so 90%+ scores can be rote replay. PRO applies reasonable perturbations and asks whether the policy still grounds the instruction.

Five official types (Figure 6 in the paper):

Object attributes — color, texture, scale; same semantic target.
Initial configuration — placements that stay physically valid. Small position shifts are where many VLAs go to zero.
Instruction — paraphrases of the same goal.
Task redesign — new object sets or composed goals. These do not mix with the other four.
Environment — background, lighting, texture; feasibility unchanged.

The other four types can be combined. Official eval matches stock LIBERO: 50 episodes per task. The paper reports models above 90% on clean LIBERO collapsing to 0% under generalized PRO settings — an upstream finding, not a hub row.

Paper: LIBERO-PRO. Project: LIBERO-PRO webpage. Code: Zxy-MLlab/LIBERO-PRO (MIT). Dataset: zhouxueyang/LIBERO-Pro.

WorldFoundry is official-results only. You run the upstream overlay, then official-validation normalizes the export. No in-tree robosuite. Base LIBERO can wire closed-loop execution; this card cannot.

Dimensions

Object attributes
KindAxis
Color, texture, or scale changes that keep the same semantic target.
Initial configuration
KindAxis
Absolute / relative object placements, still physically valid. Position shifts are where many VLAs collapse.
Instruction
KindAxis
Semantic paraphrases of the same goal. Not the 4,092-type LIBERO-Para grid.
Task redesign
KindAxis
New object sets or goals, still executable. Excluded from cross-type combinations.
Environment
KindAxis
Background, lighting, or texture context without changing feasibility.

Data composition

Size
Plug-and-play overlay on the original LIBERO suites. Five perturbation types (object, configuration, instruction, task, environment); task redesigns do not mix with the other four. Official dump: zhouxueyang/LIBERO-Pro. Protocol: 50 episodes per task, matching stock LIBERO.
Sources
zhouxueyang/LIBERO-Pro · github.com
Splits
object · configuration · instruction · task · environment
Modalities
language · vision · action · state
License
mit
Notes
The paper reports 90%+ stock-LIBERO scores collapsing to 0% under generalized PRO settings. That is an upstream finding, not a WorldFoundry board.

Metrics

Metric

success_rate
DirectionHigher is better
Episode success on a named PRO perturbation (or combination), imported from the official evaluator.
PrimarySuccess Rate
task_success
DirectionHigher is better
Per-suite or per-task success (Spatial / Object / Goal / LIBERO-10) under the declared perturbation.
episode_success
DirectionHigher is better
Per-episode success indicator when the export lists individual rollouts.
action_accuracy
DirectionHigher is better
Offline action-prediction accuracy if the official dump includes it.

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Original-split row from the paper. Task/position perturbations in LIBERO-PRO collapse to 0.00 for these models. LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization (2025).

Model	LIBERO-Goal original success
OpenVLA	0.98
π0.5	0.97
π0	0.92

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id libero-pro \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --output-dir tmp/libero-pro/official-validation \
  --json

Optional WORLDFOUNDRY_LIBERO_PRO_ROOT points at the Zxy-MLlab/LIBERO-PRO checkout. Default import path: WORLDFOUNDRY_LIBERO_PRO_RESULTS_PATH. Manifest: worldfoundry/data/benchmarks/tasks/external/libero-pro.yaml. Outputs: scorecard.json (primary success_rate), raw_results.jsonl, and evaluation/summary.json.

← Benchmark Hub