---
title: T2V-CompBench
description: "CVPR 2025 compositional T2V bench: 1,400 prompts over seven binding axes. WorldFoundry ships the runtime; it does not invent a board."
---

T2V-CompBench (HKU, with CUHK and Huawei Noah's Ark Lab) asks whether a text-to-video model **binds the right attributes, actions, and motions to the right objects**. It is a composition test, not a single aesthetic score. The official suite has **1,400 prompts** (200 × 7):

- **Consistent / dynamic attribute binding** — static properties, then temporal changes, on the correct object.
- **Spatial relationships** — 2D / 3D layout (detection + depth / SAM).
- **Motion binding** — foreground vs background motion (DOT tracking).
- **Action binding** and **object interactions** — MLLM (LLaVA-style) judges.
- **Generative numeracy** — object counts via GroundingDINO. This is the bounded official component WorldFoundry has executed on a real MP4.

Paper: [T2V-CompBench](https://arxiv.org/abs/2407.14505). Project: [t2v-compbench-2025.github.io](https://t2v-compbench-2025.github.io/). Code: [KaiyueSun98/T2V-CompBench](https://github.com/KaiyueSun98/T2V-CompBench). Reference videos: [`Kaiyue/T2V-CompBench-Videos`](https://huggingface.co/datasets/Kaiyue/T2V-CompBench-Videos) (25,200 rows).

WorldFoundry **vendors the official runtime** and a strict manifest bridge that stages generated videos into `category/NNNN.mp4`. Import official CSV / JSON, or run a category when its metric assets are staged. `t2v_compbench_average` is emitted only when all seven category scores exist.

<BenchmarkPageSections id="t2v-compbench" locale="en" />

## How to run

Import official category scores:

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id t2v-compbench \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/t2v-compbench/official-validation \
  --json
```

Recompute a category from staged videos (numeracy needs GroundingDINO + a BERT-compatible encoder):

```bash
${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/run_t2v_compbench_official_runner.py \
  --generated-video-dir <generated_videos_with_t2v_compbench_generation_manifest.jsonl> \
  --category generative_numeracy \
  --output-dir tmp/t2v-compbench/numeracy \
  --json
```

`--preflight` checks assets before a run. MLLM categories need `WORLDFOUNDRY_T2V_COMPBENCH_LLAVA_MODEL_PATH`; motion binding needs DOT; spatial categories need Depth-Anything / SAM (`WORLDFOUNDRY_T2V_COMPBENCH_SAM_CKPT`). Shared stack: `grounded_depth_segmentation_stack`. Default import path: `WORLDFOUNDRY_T2V_COMPBENCH_RESULTS_PATH`. Optional runtime root: `WORLDFOUNDRY_T2V_COMPBENCH_ROOT`.

In-tree pointers:

```text
worldfoundry/data/benchmarks/tasks/external/t2v-compbench.yaml
worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/runtime/t2v_compbench
```

Outputs include `scorecard.json` (primary `t2v_compbench_average` when complete), `raw_metric_table.jsonl`, and per-sample rows when the source exposes them.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)