WorldFoundry
English中文

WBench

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

WBench (Meituan LongCat) evaluates multi-turn interactive video world models along five axes that roll into wbench_average:

  • Quality — aesthetic, imaging, flicker, dynamics, smoothness, HPSv3
  • Setting — scene / subject adherence
  • Interaction — navigation trajectory, event edits, subject actions, perspective switches
  • Consistency — background, segment continuity, perspective, subject, geometric / photometric reconstruction, spatial (incl. gated)
  • Physical — visual plausibility, causal fidelity

Paper: WBench. Project: meituan-longcat.github.io/WBench. Code: meituan-longcat/WBench. Data: meituan-longcat/WBench. Weights: meituan-longcat/WBench-weights.

WorldFoundry checks in the metric runtime (HPSv3, TransNetV2, PAVRM, DreamSim, MegaSAM pose precompute). A real single-video temporal_flickering component is verified; the 289-case multi-turn protocol is not. --official-results-path imports report.json or per-metric directories.

Official paper teaser: WBench’s multi-turn world-generation evaluation axes.
Official paper teaser: WBench’s multi-turn world-generation evaluation axes.

Dimensions

  • Quality
    KindAxis

    Aesthetic, imaging, flicker, dynamics, smoothness, HPSv3.

  • Setting
    KindAxis

    Scene and subject adherence.

  • Interaction
    KindAxis

    Navigation, event edits, subject actions, perspective switches.

  • Consistency
    KindAxis

    Background, segment, perspective, subject, geometric/photometric, spatial.

  • Physical
    KindAxis

    Visual plausibility and causal fidelity.

Data composition

  • Size

    Meituan LongCat multi-turn video world-model bench. 28 metrics across five axes. Dataset meituan-longcat/WBench; weights meituan-longcat/WBench-weights. Official protocol cites 289 cases.

  • Sources

    meituan-longcat/WBench · meituan-longcat/WBench-examples · github.com

  • Modalities

    text · video · action

  • Notes

    A real single-video temporal_flickering component is verified. The 289-case multi-turn protocol is not.

Metrics

Metric

  • aesthetic_quality
    DirectionHigher is better

    Official WBench aesthetic quality score.

  • imaging_quality
    DirectionHigher is better

    Official WBench imaging quality score.

  • temporal_flickering
    DirectionHigher is better

    Official WBench temporal flickering score.

  • dynamic_degree
    DirectionHigher is better

    Official WBench dynamic degree score.

  • motion_smoothness
    DirectionHigher is better

    Official WBench motion smoothness score.

  • hpsv3_quality
    DirectionHigher is better

    Official WBench HPSv3 quality score.

  • scene_adherence
    DirectionHigher is better

    Official WBench scene adherence score.

  • subject_adherence
    DirectionHigher is better

    Official WBench subject adherence score.

  • navigation_trajectory
    DirectionHigher is better

    Official WBench navigation trajectory score.

  • event_edit_adherence
    DirectionHigher is better

    Official WBench event edit adherence score.

  • subject_action_adherence
    DirectionHigher is better

    Official WBench subject action adherence score.

  • perspective_switch_adherence
    DirectionHigher is better

    Official WBench perspective switch adherence score.

  • background_consistency
    DirectionHigher is better

    Official WBench background consistency score.

  • segment_continuity
    DirectionHigher is better

    Official WBench segment continuity score.

  • perspective_consistency
    DirectionHigher is better

    Official WBench perspective consistency score.

  • subject_consistency
    DirectionHigher is better

    Official WBench subject consistency cross-model score.

  • geometric_consistency
    DirectionHigher is better

    Official WBench geometric reconstruction consistency score.

  • photometric_consistency
    DirectionHigher is better

    Official WBench photometric reconstruction consistency score.

  • spatial_consistency
    DirectionHigher is better

    Official WBench spatial consistency score.

  • gated_spatial_consistency
    DirectionHigher is better

    Official WBench gated spatial consistency score.

  • visual_plausibility
    DirectionHigher is better

    Official WBench visual plausibility score.

  • causal_fidelity
    DirectionHigher is better

    Official WBench causal fidelity score.

  • quality_score
    DirectionHigher is better

    Mean WBench quality dimension score.

  • setting_score
    DirectionHigher is better

    Mean WBench setting adherence dimension score.

  • interaction_score
    DirectionHigher is better

    Mean WBench interaction dimension score.

  • consistency_score
    DirectionHigher is better

    Mean WBench consistency dimension score.

  • physical_score
    DirectionHigher is better

    Mean WBench physical dimension score.

  • wbench_average
    DirectionHigher is better

    Mean WBench score across available official dimensions.

    PrimaryWBench Average

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. meituan-longcat.github.io/WBench.

meituan-longcat.github.io · LiveOpen live board

How to run

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/wbench/run_wbench_official_runner.py \
  --run-official \
  --model-name <model_name> \
  --phase gpu \
  --metrics temporal_flickering \
  --dataset-root <WBench_dataset> \
  --generated-artifact-dir <one_officially_named_generated_video> \
  --output-dir tmp/wbench/flicker \
  --json

--submission-protocol text_multi_turn|camera_conditioned|action_conditioned only when videos truly follow that official multi-turn interface. Env: WORLDFOUNDRY_WBENCH_RESULTS_PATH, WORLDFOUNDRY_WBENCH_ROOT. Shared stack: video_world_perception_stack.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/wbench.yaml, worldfoundry/evaluation/tasks/execution/runners/wbench.

Outputs include scorecard.json (primary wbench_average).

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets