Video-Bench
Video-Bench (SJTU, with NUS, CMU, Stanford and others; CVPR 2025) scores generated video with MLLM judges aligned to human preference. Two families (official scales, then mapped 0–1 in WorldFoundry):
- Video quality (1–5) — imaging, aesthetics, temporal consistency, motion effects.
- Video-condition alignment — overall video-text consistency (1–5); object class, color, action, scene (1–3).
Few-shot scoring and chain-of-query structure the GPT-4o judge. A human-annotation dataset is released for calibration. License: Apache-2.0.
Paper: Video-Bench. Code: Video-Bench/Video-Bench. Annotations: Video-Bench/Video-Bench_human_annotation. Official videos are gated (Video-Bench_videos).
WorldFoundry vendors the runtime and wraps evaluate.py. It writes a temporary private config from env vars instead of mutating upstream config.json, and rebuilds prompt metadata from human-annotation JSONs when VideoBench_full.json is missing. Credential-gated path is present; no credentialed release run is claimed.
Dimensions
- Video quality (1–5)KindAxis
Imaging, aesthetics, temporal consistency, motion effects.
- Video-condition alignmentKindAxis
Overall video-text (1–5); object class, color, action, scene (1–3).
Data composition
- Size
CVPR 2025 human-preference-aligned MLLM video bench. GPT-4o few-shot / chain-of-query judges. Human-annotation dataset released; official videos gated. Apache-2.0.
- Sources
Video-Bench/Video-Bench_human_annotation · Video-Bench/Video-Bench_videos · github.com
- License
Apache-2.0
- Notes
WorldFoundry vendors the runtime and wraps evaluate.py. Credential-gated path is present; no credentialed release run is claimed.
Metrics
Metric
- DirectionHigher is better
imaging_qualityVideo-Bench static image quality score on the official 1-5 scale.
- DirectionHigher is better
aesthetic_qualityVideo-Bench aesthetic quality score on the official 1-5 scale.
- DirectionHigher is better
temporal_consistencyVideo-Bench temporal consistency score on the official 1-5 scale.
- DirectionHigher is better
motion_effectsVideo-Bench motion effects score on the official 1-5 scale.
- DirectionHigher is better
video_text_consistencyVideo-Bench overall video-text consistency score on the official 1-5 scale.
- DirectionHigher is better
object_class_consistencyVideo-Bench object-class consistency score on the official 1-3 scale.
- DirectionHigher is better
color_consistencyVideo-Bench color consistency score on the official 1-3 scale.
- DirectionHigher is better
action_consistencyVideo-Bench action consistency score on the official 1-3 scale.
- DirectionHigher is better
scene_consistencyVideo-Bench scene consistency score on the official 1-3 scale.
- DirectionHigher is better
videobench_averageMean normalized score across available Video-Bench dimensions.
PrimaryVideo-Bench Average
Leaderboard
Results from the paper (not a live board; not WorldFoundry-run scores). Paper video-quality table, imaging-quality column. Video-Bench: Human-Aligned Video Generation Benchmark (2025).
| Model | Imaging quality |
|---|---|
| Gen3 | 4.66 |
| Kling | 4.26 |
| VideoCrafter2 | 4.08 |
| CogVideoX | 3.87 |
| Pika-Beta | 3.78 |
| Show-1 | 3.30 |
| LaVie | 3.00 |
How to run
hf download Video-Bench/Video-Bench_human_annotation --repo-type dataset \
--local-dir cache/worldfoundry/data/hfd_datasets/Video-Bench/Video-Bench_human_annotation
worldfoundry-eval zoo benchmark-run \
--benchmark-id video-bench \
--mode official-validation \
--official-results-path <official_results.json> \
--generated-artifact-dir <generated_videos> \
--output-dir tmp/video-bench/official-validation \
--jsonRuntime judging needs WORLDFOUNDRY_VIDEOBENCH_GPT4O_API_KEY (or OPENAI_API_KEY), annotation root, and generated videos. --dimension / --mode restrict the protocol. Env: WORLDFOUNDRY_VIDEOBENCH_RESULTS_PATH, WORLDFOUNDRY_VIDEOBENCH_ANNOTATION_ROOT, WORLDFOUNDRY_VIDEOBENCH_MODEL_OUTPUT_DIR. Shared asset: video_bench_dataset_assets.
In-tree pointers: worldfoundry/data/benchmarks/tasks/external/video-bench.yaml, worldfoundry/evaluation/tasks/execution/runners/videobench/runtime.
Outputs include scorecard.json (primary videobench_average, leaderboard key overall_average).