EvalCrafter

Last updated: Oct 7, 2026

On this page

Dimensions Data composition Metrics Leaderboard How to run

EvalCrafter (Tencent AI Lab and City University of Hong Kong) is a 700-prompt text-to-video quality suite. Seventeen objective metrics roll up into four aspects plus evalcrafter_total:

Visual quality — DOVER aesthetic / technical VQA, Inception Score.
Text–video alignment — CLIP, BLIP-BLEU, SD score, detection / color / count, OCR error, celebrity-ID error.
Motion quality — action recognition, motion activity, optical flow.
Temporality — temporal CLIP, warping error, face consistency.

Paper: EvalCrafter. Project: evalcrafter.github.io. Code: evalcrafter/EvalCrafter. Prompts: RaphaelLiu/EvalCrafter_T2V_Dataset. In-tree list: worldfoundry/data/benchmarks/assets/evalcrafter/prompt700.txt.

WorldFoundry keeps an in-tree CLIP subset (clip_score, clip_temp_score, face_consistency_score) and a normalizer for official final_result.txt dumps. Videos must be prompt-index MP4s (0000.mp4 … 0699.mp4). A bundled sample_results.txt supports import smoke. Primary key: evalcrafter_total.

Official paper teaser: EvalCrafter’s large-scale video-generation evaluation suite.

Dimensions

Visual quality
KindAxis
DOVER aesthetic / technical VQA plus Inception Score.
Text–video alignment
KindAxis
CLIP, BLIP-BLEU, SD score, detection / color / count, OCR error, celebrity-ID error.
Motion quality
KindAxis
Action recognition, motion activity, and optical-flow scores.
Temporal consistency
KindAxis
Temporal CLIP, warping error, and face consistency.

Data composition

Size
700 official T2V prompts (0000.mp4 … 0699.mp4). Seventeen objective metrics roll up into four aspect scores plus Total.
Sources
RaphaelLiu/EvalCrafter_T2V_Dataset · github.com
Modalities
text · video
License
apache-2.0
Notes
In-tree official-run recomputes the CLIP-based subset (clip_score, clip_temp_score, face_consistency_score). The other 14 metrics import from official dumps.

Metrics

Metric

visual_quality
DirectionHigher is better
EvalCrafter visual-quality score.
text_video_alignment
DirectionHigher is better
EvalCrafter text-video alignment score.
motion_quality
DirectionHigher is better
EvalCrafter motion-quality score.
temporal_consistency
DirectionHigher is better
EvalCrafter temporal-consistency score.
evalcrafter_total
DirectionHigher is better
Aggregate EvalCrafter score.
PrimaryEvalCrafter Total
vqa_aesthetic
DirectionHigher is better
DOVER aesthetic component.
vqa_technical
DirectionHigher is better
DOVER technical component.
inception_score
DirectionHigher is better
EvalCrafter IS component.
clip_temp_score
DirectionHigher is better
Temporal CLIP component.
warping_error
DirectionLower is better
Temporal warping error.
face_consistency_score
DirectionHigher is better
Face consistency component.
action_score
DirectionHigher is better
Action-recognition component.
motion_ac_score
DirectionHigher is better
Motion activity component.
flow_score
DirectionHigher is better
Optical-flow component.
clip_score
DirectionHigher is better
CLIP alignment component.
blip_bleu
DirectionHigher is better
BLIP/BLEU alignment component.
sd_score
DirectionHigher is better
Stable-diffusion alignment component.
detection_score
DirectionHigher is better
Detection alignment component.
color_score
DirectionHigher is better
Color alignment component.
count_score
DirectionHigher is better
Object-count alignment component.
ocr_error
DirectionLower is better
OCR mismatch component.
celebrity_id_error
DirectionLower is better
Celebrity identity mismatch component.

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. evalcrafter.github.io.

evalcrafter.github.io · LiveOpen live board

How to run

Import official aggregates (smoke uses the bundled sample):

worldfoundry-eval zoo benchmark-run \
  --benchmark-id evalcrafter \
  --mode official-validation \
  --official-results-path worldfoundry/data/benchmarks/assets/evalcrafter/sample_results.txt \
  --output-dir tmp/evalcrafter/official-validation \
  --json

Recompute the in-tree CLIP subset from numbered MP4s:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/evalcrafter/generated_videos
worldfoundry-eval zoo benchmark-run \
  --benchmark-id evalcrafter \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/evalcrafter/official-run \
  --json

CLIP ViT-B/32 resolves through WORLDFOUNDRY_CLIP_VIT_B32_MODEL_DIR or --clip-model. Default import path: WORLDFOUNDRY_EVALCRAFTER_RESULTS_PATH. Optional root: WORLDFOUNDRY_EVALCRAFTER_ROOT.

In-tree pointers:

worldfoundry/data/benchmarks/tasks/external/evalcrafter.yaml
worldfoundry/evaluation/tasks/execution/runners/evalcrafter/
worldfoundry/data/benchmarks/assets/evalcrafter

Outputs include scorecard.json (primary evalcrafter_total), benchmark_contract.json, and raw_metric_table.jsonl.

← Benchmark Hub