---
title: EvalCrafter
description: "700-prompt T2V quality bench: visual, alignment, motion, temporality (17 metrics → Total). WorldFoundry recomputes CLIP subset or imports; no invented board."
---

EvalCrafter (Tencent AI Lab and City University of Hong Kong) is a **700-prompt** text-to-video quality suite. Seventeen objective metrics roll up into four aspects plus `evalcrafter_total`:

- **Visual quality** — DOVER aesthetic / technical VQA, Inception Score.
- **Text–video alignment** — CLIP, BLIP-BLEU, SD score, detection / color / count, OCR error, celebrity-ID error.
- **Motion quality** — action recognition, motion activity, optical flow.
- **Temporality** — temporal CLIP, warping error, face consistency.

Paper: [EvalCrafter](https://arxiv.org/abs/2310.11440). Project: [evalcrafter.github.io](http://evalcrafter.github.io). Code: [evalcrafter/EvalCrafter](https://github.com/evalcrafter/EvalCrafter). Prompts: [`RaphaelLiu/EvalCrafter_T2V_Dataset`](https://huggingface.co/datasets/RaphaelLiu/EvalCrafter_T2V_Dataset). In-tree list: `worldfoundry/data/benchmarks/assets/evalcrafter/prompt700.txt`.

WorldFoundry keeps an **in-tree CLIP subset** (`clip_score`, `clip_temp_score`, `face_consistency_score`) and a normalizer for official `final_result.txt` dumps. Videos must be prompt-index MP4s (`0000.mp4` … `0699.mp4`). A bundled `sample_results.txt` supports import smoke. Primary key: `evalcrafter_total`.

<BenchmarkPageSections id="evalcrafter" locale="en" />

## How to run

Import official aggregates (smoke uses the bundled sample):

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id evalcrafter \
  --mode official-validation \
  --official-results-path worldfoundry/data/benchmarks/assets/evalcrafter/sample_results.txt \
  --output-dir tmp/evalcrafter/official-validation \
  --json
```

Recompute the in-tree CLIP subset from numbered MP4s:

```bash
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/evalcrafter/generated_videos
worldfoundry-eval zoo benchmark-run \
  --benchmark-id evalcrafter \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/evalcrafter/official-run \
  --json
```

CLIP ViT-B/32 resolves through `WORLDFOUNDRY_CLIP_VIT_B32_MODEL_DIR` or `--clip-model`. Default import path: `WORLDFOUNDRY_EVALCRAFTER_RESULTS_PATH`. Optional root: `WORLDFOUNDRY_EVALCRAFTER_ROOT`.

In-tree pointers:

```text
worldfoundry/data/benchmarks/tasks/external/evalcrafter.yaml
worldfoundry/evaluation/tasks/execution/runners/evalcrafter/
worldfoundry/data/benchmarks/assets/evalcrafter
```

Outputs include `scorecard.json` (primary `evalcrafter_total`), `benchmark_contract.json`, and `raw_metric_table.jsonl`.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)