---
title: WorldBench
description: "UCLA diagnostic physics bench: 425 Kubric scenes, SAM2 vs ground-truth physics, plus text QA. WorldFoundry implements an in-tree scorer; it does not invent a board."
---

WorldBench (UCLA, with Sony AI, Yale, and DEVCOM ARL) diagnoses how close world models are to **physics**. **Kubric** (PyBullet + Blender) renders **425** physically accurate scenes across four concepts — motion physics, object permanence, support relations, scale / perspective — with ground-truth depth, normals, segmentation, and flow. Models generate continuations from initial frames; **SAM2** tracks first-frame boxes and masks are compared to simulation (foreground mIoU, background RMSE). A textual subset adds multiple-choice and yes/no QA.

The public evaluator calls Dice “mIoU”. WorldFoundry reports **conventional `foreground_miou`** and the release’s **`foreground_dice` compatibility value** separately. The inspected HF tree currently has **375** scene directories (15 scenarios × 25) versus the paper’s 425.

Paper: [WorldBench](https://arxiv.org/abs/2601.21282). Project: [world-bench.github.io](https://world-bench.github.io/). Data: [`worldbenchmark/IntuitivePhysics`](https://huggingface.co/datasets/worldbenchmark/IntuitivePhysics). Dataset and evaluator declare **no redistribution license**.

WorldFoundry implements an in-tree, model-backed runner. Runtime environments still record: **no confirmed redistributable official raw-video evaluator**, and **`official-run` from videos is not a release claim**. Import via `WORLDFOUNDRY_WORLDBENCH_RESULTS_PATH` is the supported compatibility path.

<BenchmarkPageSections id="worldbench" locale="en" />

## How to run

```bash
hf download worldbenchmark/IntuitivePhysics --repo-type dataset \
  --local-dir cache/worldfoundry/data/hfd_datasets/worldbenchmark/IntuitivePhysics

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/worldbench/run_worldbench_official_runner.py \
  --run-official \
  --video-only \
  --dataset-root "${WORLDFOUNDRY_WORLDBENCH_DATASET_ROOT}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --sam2-checkpoint "${WORLDFOUNDRY_WORLDBENCH_SAM2_CKPT}" \
  --output-dir tmp/worldbench/video \
  --json
```

`--predicted-mask-dir` skips SAM2 when masks already exist. `--text-only` plus `--answer-manifest` covers QA. Frame knobs: `--ground-truth-start-frame`, `--generated-skip-frames`. Shared assets: `worldbench_dataset_assets`, `facebook/sam2.1-hiera-large`.

In-tree pointers: `worldfoundry/data/benchmarks/tasks/external/worldbench.yaml`, `worldfoundry/evaluation/tasks/execution/runners/worldbench`.

Outputs include `scorecard.json` (primary `foreground_miou`) and `worldbench_evaluation.json`.

## Dataset, videos, and frame alignment

The in-tree runtime does **not** import an upstream checkout. SAM2 comes from `worldfoundry.base_models.perception_core.segment.sam2`. Dataset root must be the public HF layout:

```text
<dataset>/
  scenes/<group>/<scenario>/<scene>/
    rgba_00000.png
    segmentation_00000.png
    …
  textual_questions/*.json
```

Generated continuations can mirror scene ids:

```text
<generated>/<group>/<scenario>/<scene>.mp4
<generated>/<group>/<scenario>/<scene>/output.mp4
<generated>/<group>/<scenario>/<scene>/00000.png
```

Any other video layout needs `--video-manifest` (JSON / JSONL / CSV / TSV) with `sample_id` and `video_path`. Text predictions use `--answer-manifest`. Stable `question_id` is `<question-file-stem>:<row>`; or include both `video_name` and `question` because some videos have two questions.

The bench uses **nine conditioning frames**. By default, generated frame 0 aligns to GT frame 9 and **24 continuation frames** are scored. If the model prepends those nine frames, pass `--generated-skip-frames 9`. Defaults live in `worldfoundry/data/benchmarks/assets/worldbench/evaluator.yaml`.

Raw video scoring defaults to `facebook/sam2.1-hiera-large`. Set `WORLDFOUNDRY_WORLDBENCH_SAM2_CKPT` to a local file, otherwise the HF cache resolver is used. Import-only path: `--official-results-path` without `--run-official`.

Use `worldfoundry-unified-cu128`. A successful in-tree run is integration evidence, not `leaderboard_valid`, until full-dataset numerical parity is audited. The inspected upstream pin `WorldBenchmark/worldbench_eval@4575a164…` has no license file and a broken public evaluator; WorldFoundry keeps provenance in the catalog YAML, not a `SOURCE.json`.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)