VideoScore

VideoScore (TIGER-Lab, University of Waterloo) is a learned reward model, not a prompt suite. It is trained on VideoFeedback (37.6K human-annotated videos) and predicts five aspect scores (official 1–4 scale) for any generated clip:

• Visual quality
• Temporal consistency
• Dynamic degree
• Text-to-video alignment
• Factual consistency

videoscore_average is the mean of the five. VideoScore-Bench evaluates scorer–human correlation (SPCC / pairwise). License: MIT (runtime); dataset stacks Apache-2.0 / MIT.

Paper: VideoScore. Project: tiger-ai-lab.github.io/VideoScore. Code: TIGER-AI-Lab/VideoScore. Bench data: TIGER-Lab/VideoScore-Bench.

WorldFoundry vendors the runtime, supports bounded in-process GPU scoring, and wraps benchmark/eval_videoscore.py. A bounded official scorer GPU run is verified; SPCC / pairwise full-suite parity is not.

Dimensions

• Five-aspect reward model

  KindAxis

  Visual quality, temporal consistency, dynamic degree, T2V alignment, factual consistency on a 1–4 scale.

• VideoScore-Bench

  KindSuite

  Scorer–human correlation (SPCC / pairwise) over VideoFeedback-style rows.

Data composition

• Size

  TIGER-Lab learned reward model trained on VideoFeedback (37.6K human-annotated videos). Not a prompt suite. MIT runtime; dataset stacks Apache-2.0/MIT.

• Sources

  TIGER-Lab/VideoFeedback · TIGER-Lab/VideoScore-Bench · github.com

• License

  MIT

• Notes

  A bounded official scorer GPU run is verified. SPCC/pairwise full-suite parity is not.

Metrics

Metric

• visual_quality

  DirectionHigher is better

  VideoScore visual quality aspect score on a 1-4 scale.

• temporal_consistency

  DirectionHigher is better

  VideoScore temporal consistency aspect score on a 1-4 scale.

• dynamic_degree

  DirectionHigher is better

  VideoScore dynamic-degree aspect score on a 1-4 scale.

• text_to_video_alignment

  DirectionHigher is better

  VideoScore text-to-video alignment aspect score on a 1-4 scale.

• factual_consistency

  DirectionHigher is better

  VideoScore factual consistency aspect score on a 1-4 scale.

• videoscore_average

  DirectionHigher is better

  Mean of the five VideoScore aspect scores.

  PrimaryVideoScore Average

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Judge–human correlation from the paper, not a generator quality board. VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation (2024).

Model	Spearman average (×100)
VideoScore (gen)paper method	77.1
VideoScore (reg)paper method	75.7
GPT-4o	23.0
Gemini-1.5-Flash	20.8
Gemini-1.5-Pro	16.9

How to run

hf download TIGER-Lab/VideoScore-Bench --repo-type dataset --local-dir /path/to/VideoScore-Bench
export WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT=/path/to/VideoScore-Bench

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py \
  --bench-name video_feedback \
  --bounded-sample-count 4 \
  --dataset-root "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" \
  --output-dir tmp/videoscore/bounded \
  --json

Import upstream eval_*_videoscore.json via --official-results-path / --from-upstream-results. Env: WORLDFOUNDRY_VIDEOSCORE_MODEL_DIR, WORLDFOUNDRY_VIDEOSCORE_FRAMES_DIR, WORLDFOUNDRY_VIDEOSCORE_ROOT. Shared stack: videoscore_reward_metric_stack.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/videoscore.yaml, worldfoundry/evaluation/tasks/execution/runners/videoscore/runtime/videoscore.

Outputs include scorecard.json (primary videoscore_average).

← Benchmark Hub