WorldFoundry
English中文

WorldReasonBench

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

WorldReasonBench (Tsinghua, with NTU and Waterloo) reframes video-generation eval as world-state prediction: given an initial state and an action, can the model generate a future whose evolution stays physically, socially, logically, and informationally consistent? Official size: 436 curated cases with structured QA across four reasoning dimensions / 22 subcategories. Companion WorldRewardBench adds ~6K expert preference pairs.

Two protocols:

  • Process-aware Reasoning Verification — structured QA plus reasoning-phase diagnostics. Primary score_pr geometrically combines QA accuracy with dynamic reasoning. reasoning_gap (static outcome minus dynamic reasoning; lower is better) flags outcome hacking.
  • Multi-dimensional Quality Assessment — reasoning quality, temporal consistency, visual aesthetics. Reward-model path also reports point-wise / pairwise correlations.

Paper: WorldReasonBench. Project: unix-ai-lab.github.io/WorldReasonBench. Code: UniX-AI-Lab/WorldReasonBench. Upstream README prohibits redistribution, modification, and commercial use without approval despite also referring to MIT.

WorldFoundry hosts the protocol runtime with an OpenAI-compatible multimodal judge. Result normalization / QA component code exists; no complete official judge run is claimed.

Official paper teaser: video generators scored as future world-state predictors.
Official paper teaser: video generators scored as future world-state predictors.

Dimensions

  • Process-aware Reasoning Verification
    KindTrack

    Structured QA plus reasoning-phase diagnostics. Primary score_pr. reasoning_gap lower-is-better.

  • Multi-dimensional Quality Assessment
    KindTrack

    Reasoning quality, temporal consistency, visual aesthetics.

  • WorldRewardBench
    KindSuite

    ~6K expert preference pairs for point-wise/pair-wise reward-model eval.

Data composition

  • Size

    Tsinghua world-state prediction bench. 436 cases, 4 reasoning dimensions / 22 subcategories. Restricted: README prohibits redistribution/modification/commercial use without approval.

  • Sources

    WorldReasonBench · github.com

  • Modalities

    image · video · text

  • License

    restricted_noncommercial

  • Notes

    Result normalization/QA component code exists. No complete official judge run is claimed.

Metrics

Metric

  • score_pr
    DirectionHigher is better

    Geometric combination of QA accuracy and dynamic reasoning score.

    PrimaryProcess-aware Reasoning Score

  • qa_accuracy
    DirectionHigher is better

    Structured QA accuracy over ground-truth world-state questions.

  • state_score
    DirectionHigher is better

    Correctness of the final predicted world state.

  • process_score
    DirectionHigher is better

    Reasoning-phase diagnostic score over the state-evolution process.

  • fidelity_score
    DirectionHigher is better

    Content fidelity of the generated evolution.

  • mechanism_score
    DirectionHigher is better

    Correctness of the causal mechanism driving the evolution.

  • static_outcome_score
    DirectionHigher is better

    Outcome score judged from static (final-state) evidence only.

  • dynamic_reasoning_score
    DirectionHigher is better

    Reasoning score judged from the full temporal evolution.

  • reasoning_gap
    DirectionLower is better

    Static outcome score minus dynamic reasoning score; lower values indicate less outcome hacking.

  • pointwise_score
    DirectionHigher is better

    Point-wise quality score used for reward-model evaluation.

  • reasoning_correctness
    DirectionHigher is better

    Multi-dimensional assessment: reasoning quality.

  • content_fidelity
    DirectionHigher is better

    Multi-dimensional assessment: temporal/content consistency.

  • visual_aesthetics
    DirectionHigher is better

    Multi-dimensional assessment: visual aesthetics.

  • pointwise_spearman
    DirectionHigher is better

    Spearman correlation of point-wise scores with expert rankings.

  • induced_pairwise_accuracy
    DirectionHigher is better

    Pairwise preference accuracy induced from point-wise scores.

  • pairwise_accuracy_with_ties
    DirectionHigher is better

    Pairwise preference accuracy counting tied pairs.

  • pairwise_accuracy_without_ties
    DirectionHigher is better

    Pairwise preference accuracy excluding tied pairs.

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors (2026).

ModelScore PR (%)
Seedance2.039.8
Veo3.1-Fast35.3
Sora2-8s34.3
Kling32.7
Wan2.632.4
Sora2-12s32.4
HunyuanVideo-1.517.9
Wan2.2-14B17.5

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id worldreasonbench \
  --mode official-validation \
  --official-results-path <official_results_dir> \
  --output-dir tmp/worldreasonbench/official-validation \
  --json

--protocol selects the protocol. QA / pair files: --qa-json, --pairs-json. Judge: --base-url, --api-key, --model, --judge-model. Default import: WORLDFOUNDRY_WORLDREASONBENCH_RESULTS_PATH.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/worldreasonbench.yaml, worldfoundry/evaluation/tasks/execution/runners/worldreasonbench.

Outputs include scorecard.json (primary score_pr) and protocol_results.json.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets