WorldFoundry
English中文

IPV-Bench

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

IPV-Bench (Impossible Videos; Show Lab, NUS) flips the usual physics question. It asks whether models can depict counterfactual, physically impossible events — and whether video-understanding models can recognize them. The taxonomy covers physical, biological, geographical, and social impossibilities. Two tracks:

  • Generation (IPV-Txt) — visual quality and prompt following; impossible_video_score is the share of clips that are both high-quality and prompt-faithful.
  • Understanding (IPV-Vid) — judgement, multi-choice QA, and open QA against official labels.

ipv_bench_average aggregates whatever official families you supply.

Paper: Impossible Videos. Code: showlab/Impossible-Videos.

WorldFoundry normalizes official CSV/JSON exports for both tracks. It does not run the official judges or understanding models in tree. --run-fixture is wiring only.

Official paper teaser: impossible-video examples that break physical or commonsense rules.
Official paper teaser: impossible-video examples that break physical or commonsense rules.

Dimensions

  • IPV-Txt generation
    KindTrack

    Visual quality and prompt following; impossible_video_score is the share of clips that are both high-quality and prompt-faithful.

  • IPV-Vid understanding
    KindTrack

    Judgement, multi-choice QA, and open QA against official labels.

Data composition

  • Size

    NUS Show Lab Impossible Videos: generate and recognize physically, biologically, geographically, and socially impossible events. Two tracks (IPV-Txt / IPV-Vid).

  • Sources

    github.com

  • Modalities

    text · video

  • Notes

    WorldFoundry normalizes official CSV/JSON. It does not run the official judges or understanding models.

Metrics

Metric

  • visual_quality
    DirectionHigher is better

    IPV-Bench human or official judge annotation for generated-video visual quality.

  • prompt_following
    DirectionHigher is better

    IPV-Bench human or official judge annotation for prompt following.

  • impossible_video_score
    DirectionHigher is better

    Percentage of generated videos that are both high quality and good prompt following under the IPV-Bench protocol.

  • judgement_accuracy
    DirectionHigher is better

    IPV-Bench video-understanding judgement task accuracy against official labels.

  • mcqa_accuracy
    DirectionHigher is better

    IPV-Bench multi-choice QA accuracy against official labels.

  • open_qa_score
    DirectionHigher is better

    IPV-Bench open-ended QA score from official evaluator or judge outputs.

  • ipv_bench_average
    DirectionHigher is better

    IPV-Bench aggregate when supplied by official annotations, judge outputs, or normalized result files.

    PrimaryIPV-Bench Average

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Impossible Videos (2025).

ModelIPV score
Mochi 137.3
Pyramid-Flow19.6
CogVideoX-1.516.9
Open-Sora15.8
LTX10.0

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id ipv-bench \
  --mode official-validation \
  --official-results-path <official_results.csv> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/ipv-bench/official-validation \
  --json

Default import: WORLDFOUNDRY_IPV_BENCH_RESULTS_PATH. Optional root / backend: WORLDFOUNDRY_IPV_BENCH_ROOT, WORLDFOUNDRY_IPV_BENCH_RUNTIME_BACKEND. No Hugging Face dataset is required by the contract.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/ipv-bench.yaml, worldfoundry/evaluation/tasks/execution/runners/ipv_bench.

Outputs include scorecard.json (primary ipv_bench_average) and raw_metric_table.jsonl.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets