Benchmark Hub

Pick a benchmark by what it actually measures, then open its manifest-backed recipe — import, score, or closed-loop — without invented rankings.

Edit this page GitHub Markdown

Last updated: Oct 7, 2026

On this page

How to pick a bench What a recipe records How to run one benchmark

This hub is a recipe index, not a leaderboard site. Each card is generated from benchmark catalogs and task manifests: what the suite measures, which files it needs, and which WorldFoundry path is real (official import, in-tree scoring, or closed-loop). It does not invent ranks. Detail pages link a live author-maintained board when one exists, otherwise they cite the paper table or say so.

74 benchmarkscatalog data from repository manifests

How to pick a bench

Start from the claim you need to test, then open that recipe — do not install a dedicated environment first.

Lifelong / knowledge-transfer manipulation — LIBERO (four suites on a Franka tabletop). Related kitchen work is RoboCasa.
GPU-parallel skill families — ManiSkill (SAPIEN). WorldFoundry imports rollouts; it does not replace the simulator.
Long-horizon language chains — CALVIN (A/B/C → D, success at chain length 1–5).
Real-to-sim policy ranking — SimplerEnv (Google Robot and WidowX/Bridge, calibrated to real setups).
Household activities with logical goals — BEHAVIOR-1K (1,000 BDDL activities in OmniGibson).
Whether a video model continues real physics — Physics-IQ (5-second continuations; in-tree scorer). Pair with Physics-IQ Verified only when that protocol is the one you mean — the media are not interchangeable.
Open T2V clip quality / composition — VBench, T2V-CompBench, FETV.

Before installing benchmark-specific packages, read the runtime environment matrix. It separates the unified environment, managed dedicated environments, hosted judges, and dimension-specific native/checkpoint stacks.

What a recipe records

Dimensions: Tasks, axes, suites, tracks, or splits the catalog actually records — not marketing tags.
Data composition: Prompt or task count, dataset sources, modalities, and license when those facts are recorded.
Metrics: Scorecard keys, direction, and the primary aggregate used for the run.
Leaderboard: Live author-maintained boards when they exist, otherwise cited paper tables. Catalog inclusion is not a WorldFoundry ranking.
How to run: Official import, in-tree scoring, checkpoints, and environment boundaries.

How to run one benchmark

Open the recipe first. Prompt files, generated-artifact layouts, checkpoint names, judge models, and official result formats are benchmark-specific.

Import official result files

Use this when the recipe says WorldFoundry normalizes an upstream export and does not run the simulator.

worldfoundry-eval zoo benchmark-run \
  --benchmark-id <benchmark-id> \
  --mode official-validation \
  --official-results-path <official-result-file-or-dir> \
  --generated-artifact-dir <generated-artifact-dir> \
  --output-dir tmp/<benchmark-id>/official-validation \
  --json

Score generated artifacts in tree

Use official-run only when the recipe says the in-tree runtime can recompute scores from generated artifacts and the required media or checkpoints are staged. Physics-IQ is the usual example.

worldfoundry-eval zoo benchmark-run \
  --benchmark-id <benchmark-id> \
  --mode official-run \
  --generated-artifact-dir <generated-artifact-dir> \
  --output-dir tmp/<benchmark-id>/official-run \
  --json

To generate artifacts with a registered model before scoring:

worldfoundry-eval run \
  --benchmark <benchmark-id> \
  --model <model-id> \
  --mode official-run \
  --output-dir tmp/<benchmark-id>/<model-id> \
  --json