WorldFoundry
English中文

VideoVerse

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

VideoVerse (OPPO Research Institute, with Sun Yat-sen and HK PolyU) asks how far a text-to-video generator is from a world model. Official size: 300 prompts, each decomposed into ordered events with binary evaluation questions across ten dimensions. Families include QA accuracy, event coverage, temporal causality, world-knowledge consistency, and static / dynamic scene consistency. Judging combines event-LCS over detected event order with per-question verification.

Paper: VideoVerse. Project: naptmn.cn/Homepage_of_VideoVerse. Code: Zeqing-Wang/VideoVerse. License: Apache-2.0.

WorldFoundry runs the judge in tree (Gemini default, URL, or local VLM) and normalizes eval_res.json through official event-LCS / verification logic. Prompt manifests are bundled. --strict enforces the canonical 300-prompt suite. Official path is integrated; a full credentialed suite is not release-verified.

Official paper teaser: VideoVerse’s world-model evaluation dimensions for T2V.
Official paper teaser: VideoVerse’s world-model evaluation dimensions for T2V.

Dimensions

  • Event-decomposed prompts
    KindSuite

    300 prompts decomposed into ordered events with binary QA across ten dimensions.

Data composition

  • Size

    OPPO 300-prompt world-model T2V bench. Event-LCS plus per-question verification. Prompt manifests bundled. Apache-2.0.

  • Sources

    NNaptmn/VideoVerse · github.com

  • Modalities

    text · video

  • License

    Apache-2.0

  • Notes

    WorldFoundry runs Gemini/URL/local VLM judges. Official path is integrated; a full credentialed suite is not release-verified.

Metrics

Metric

  • qa_accuracy
    DirectionHigher is better

    VideoVerse binary/question-answer judging accuracy.

  • event_coverage
    DirectionHigher is better

    Coverage of required events.

  • temporal_causality
    DirectionHigher is better

    Temporal causality consistency score.

  • world_knowledge_consistency
    DirectionHigher is better

    World-knowledge consistency score.

  • static_scene_consistency
    DirectionHigher is better

    Static scene consistency score.

  • dynamic_event_consistency
    DirectionHigher is better

    Dynamic event consistency score.

  • videoverse_average
    DirectionHigher is better

    Mean over available VideoVerse score families.

    PrimaryVideoVerse Average

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Paper Table 1 overall. Sora-2 missed 4 videos due to safety review. VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos? (2025).

ModelOverall (Gemini 2.5 Pro judge)
Sora-24 videos missing1299
Veo-31292
Minimax-Hailuo1203
Wan2.2-A14B1085
OpenSora2.0989
Wan2.1-14B969
SkyReels-V2 (L)968
Hunyuan898

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id videoverse \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/videoverse/official-validation \
  --json

--run-official judges generated videos against bundled prompts_of_VideoVerse.json / decomposed variant (GEMINI_API_KEY or WORLDFOUNDRY_VIDEOVERSE_GEMINI_API_KEY). Use --prompt-manifest only for non-standard assets. Default import: WORLDFOUNDRY_VIDEOVERSE_RESULTS_PATH. Shared asset: videoverse_dataset_assets.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/videoverse.yaml, worldfoundry/data/benchmarks/assets/videoverse/prompt/.

Outputs include scorecard.json (primary videoverse_average), eval_res.json, and judge_responses.jsonl.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets