PAWBench

PAWBench (SJTU, Shanghai AI Lab, Krea AI, Hugging Face, Shanghai Innovation Institute, Tongyi Lab, HKU) asks whether repeated generations recover the distribution of physically valid futures — not whether one clip looks plausible. Official size: 50 scenes (25 calibration with reference outcome probabilities, 25 coverage). A complete run generates 50 independent rollouts per scene = 2,500 videos.

The two tracks stay separate. Upstream defines no combined scalar.

• Calibration — calibration_tvd_percent (lower is better): total-variation distance to the reference outcome distribution.
• Coverage — coverage_percent: share of the official physical-outcome support seen across rollouts.

Paper: PAWBench. Project: pawbench.github.io. Code: Andrew0613/PAWBench (Apache-2.0). Data: Andrew613/PAWBench.

WorldFoundry vendors the official PAWEval judge. Fresh judging defaults to Gemini 3.5 Flash through OpenRouter; other OpenAI-compatible judges are analysis-only. Existing metrics.json or run directories import without network.

Dimensions

• PAW-Calibration

  KindTrack

  25 scenes with reference outcome probabilities. Metric: calibration_tvd_percent (lower is better).

• PAW-Coverage

  KindTrack

  25 coverage scenes. Metric: coverage_percent.

Data composition

• Size

  50 scenes (25 calibration + 25 coverage). Official run: 50 independent rollouts per scene = 2,500 videos. No official combined scalar. Apache-2.0.

• Sources

  Andrew613/PAWBench · github.com

• Modalities

  image · video · text

• License

  Apache-2.0

• Notes

  WorldFoundry vendors PAWEval (Gemini 3.5 Flash via OpenRouter by default). Incomplete coverage is a blocker, not a silent average.

Metrics

Metric

• calibration_tvd_percent

  DirectionLower is better

  Total variation distance between generated and reference outcome distributions, reported as a percentage.

• coverage_percent

  DirectionHigher is better

  Percentage of the official physical-outcome support observed across repeated generated rollouts.

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Paper calibration TVD. Coverage is reported separately. PAWBench: How Far Are We from Probabilistically Aligned World Modeling? (2026).

Model	PAW-Calibration TVD ×100 (lower is better)
Cosmos 3 Super I2V	20.5
MiniMax H3	24.2
Wan2.7	26.3
Wan2.2	26.3
LTX-2.3	30.1
LTX-2.5	30.2
Seedance 2	30.5
Kling 3 Std.	34.9

How to run

hf download Andrew613/PAWBench --repo-type dataset --local-dir /data/pawbench
export OPENROUTER_API_KEY=<your-key>

worldfoundry-eval zoo benchmark-run \
  --benchmark-id pawbench \
  --mode official-run \
  --benchmark-data-root /data/pawbench \
  --generated-artifact-dir /data/my-model-rollouts \
  --output-dir tmp/pawbench/official-run \
  --json

Import a prepared run:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id pawbench \
  --mode official-validation \
  --official-results-path /path/to/pawbench/run \
  --output-dir tmp/pawbench/imported \
  --json

Outputs include scorecard.json, raw_metric_table.jsonl, per_sample_scores.jsonl, and benchmark_contract.json. The checked-in sample is a normalizer fixture only.

Rollouts, judge, and metrics.json

Workflow: benchmark package → your video generator → rollout directory → PAWEval → metrics.json. The generator and the judge are different models. The official runner never loads the video model; it scores videos already on disk. Extra pip pins: worldfoundry/data/benchmarks/requirements/pawbench.txt.

The HF download is the scene table, source images, generation prompts, and scoring policy (100 reviewable rubrics: one outcome + one trustworthiness rubric per scene). Write the complete 50 × 50 grid as:

my-model-rollouts/
  <scene-id>/
    r000.mp4
    …
    r049.mp4

Official PAWEval uses Gemini 3.5 Flash through OpenRouter (OPENROUTER_API_KEY). OPENAI_API_KEY is not part of that path. Other OpenAI-compatible judges (--vlm-base-url, --vlm-model, --vlm-api-key-env) are analysis-only and are not comparable to the official table.

--model-name is the label written into result files; it does not load a checkpoint. Re-running the same benchmark, model name, videos, and judge resumes finished rollout judgments.

PAWEval writes run.json, checkpoint.jsonl, rows.jsonl, and metrics.json under the run output. A complete run has status: "ok" and no blockers. Track averages stay separate:

• Calibration TVD: tracks.calibration.models.<model>.track_average (lower is better).
• Coverage: tracks.coverage.models.<model>.track_average (higher is better).

If status is "blocked", fix the listed missing-video / judge / infrastructure errors before reporting. Trustworthiness rubrics record action, continuity, and process failures without changing the terminal outcome label.

One rollout is a readout, not a model score. Across 50 coin-flip rollouts, Head/Tail frequencies are compared to the scene reference with TVD.

← Benchmark Hub