---
title: WRBench
description: "USTC / X-Humanoid persistent-state bench: Natural-25, D1–D6 camera look-away chain. WorldFoundry checks in the package; full D1–D6 is unverified."
---

WRBench (“Current World Models Lack a Persistent State Core”; USTC and X-Humanoid / Beijing Innovation Center of Humanoid Robotics) treats camera motion as an **observability intervention**: an event starts, the camera looks away, and on return the bench asks whether the target **advanced to the endpoint or froze**. **Natural-25** crosses 25 scene families with a four-level event design (spatial change × state change). The canonical camera track expands **100 semantic variants × five camera controls = 500 requests**.

Hierarchical **D1–D6**:

- **D1** — requested-camera precision / prompt-camera alignment
- **D2** — visual integrity
- **D3 / D4** — visible spatial / state consistency
- **D5 / D6** — re-observed spatial / state consistency (aggregated **only** over judgeable hidden-and-returned evidence)

`wrbench_average` aggregates D1–D6. Paper: [WRBench](https://arxiv.org/abs/2606.20545). Project: [jinplu.github.io/WRBench](https://jinplu.github.io/WRBench/). Code: [JinPLu/WRBench](https://github.com/JinPLu/WRBench). Data: [`WRBench/wrbench-natural25`](https://huggingface.co/datasets/WRBench/wrbench-natural25).

WorldFoundry **checks in the complete package and Natural-25 assets**. Natural-25 request materialization and result normalization are available; **full D1–D6 execution is unverified**. Scorers: VGGT-Omega, DINOv2, Qwen3.5, Qwen3-VL.

<BenchmarkPageSections id="wrbench" locale="en" />

## How to run

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id wrbench \
  --mode official-validation \
  --official-results-path <official_results.json_or_csv> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/wrbench/official-validation \
  --json
```

`--run-official` scores generated videos plus camera sidecars against Natural-25 (`--scorer-profile`, `--sidecar-profile-gate`). Rows without re-observation support are **excluded** from D5/D6, not counted as failures. Env: `WORLDFOUNDRY_WRBENCH_RESULTS_PATH`, `WORLDFOUNDRY_WRBENCH_ROOT`. Shared stack: `vggt_omega`, `dinov2`, `qwen3_5_vl`, `qwen3_vl`.

In-tree pointers: `worldfoundry/data/benchmarks/tasks/external/wrbench.yaml`, `worldfoundry/data/benchmarks/assets/wrbench`, `worldfoundry/evaluation/tasks/execution/runners/wrbench/runtime/wrbench`.

Outputs include `scorecard.json` (primary `wrbench_average`, leaderboard key `Avg`) and `generation_requests.jsonl`.

## Bundled Natural-25 assets

`worldfoundry/data/benchmarks/assets/wrbench/natural25/` ships the prompt grid WorldFoundry materializes (Apache-2.0 with the rest of this repository). Hugging Face mirrors: [collection](https://huggingface.co/collections/WRBench/wrbench-current-world-models-lack-a-persistent-state-core-6a365c717251293c9fc2cc26), [`WRBench/wrbench-natural25`](https://huggingface.co/datasets/WRBench/wrbench-natural25), [`WRBench/wrbench-videos`](https://huggingface.co/datasets/WRBench/wrbench-videos), [leaderboard](https://huggingface.co/spaces/WRBench/wrbench-leaderboard).

| File | Role |
| --- | --- |
| `scene_events_25x4.csv` | 25 scene families × 4 event axes (none / spatial / state / full) |
| `families.jsonl` | Scene-family metadata and prompt variants |
| `variants.jsonl` | Deterministic TI2V variants: 25 families × 4 event tiers × 4 camera gaps. `ti2v_prompt` is the first-frame-anchored prompt of record |
| `variants.legacy_pronoun_20260620.jsonl` | Frozen provenance snapshot; currently byte-identical to `variants.jsonl` |
| `t2v_layout_anchors.jsonl` | Text-only layout facts (subject left, interactor right, open surface, background) |
| `t2v_event_tails.jsonl` | Text-only event/action tails keyed by `variant_id` |
| `prompt_profiles/t2v_layout_anchor.json` | Text-only prompt profile; bans T2I/TI2V style tokens |
| `camera_scopes/t2v_rotation_stress_30_60.json` | T2V-only rotation-stress scope (`static`, `yaw30_LR`, `yaw30_RL`, `yaw60_LR`, `yaw60_RL`) |
| `first_frames/` + `first_frames_manifest.json` | Released PNG first frames, one per `family_id` |

You can regenerate first-frame PNGs with `wrbench firstframe` or substitute your own T2I images. Human annotation verdicts are released separately. Text-only leaderboard entries should materialize `t2v_layout_anchor` and the rotation-stress scope rather than the main TI2V/TV2V surface.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)