FETV

FETV (Peking University and Huawei Noah's Ark Lab) is a fine-grained open-domain text-to-video bench. 619 prompts sit on three orthogonal axes: major content, attribute control (color, count, style), and prompt complexity (simple vs compositional). Scoring mixes human ratings (static quality, temporal quality, overall / fine-grained alignment) with automatic metrics (CLIPScore, BLIPScore, FID, FVD). fetv_average is emitted only when every declared component is present.

Paper: FETV (NeurIPS 2023 D&B). Prompts: llyx97/FETV. Scorer: llyx97/FETV-EVAL. Dataset: lyx97/FETV (CC-BY-4.0).

WorldFoundry vendors FETV-EVAL, joins generated videos to the official prompt index (official 16-frame layout), runs the CLIPScore component through shared base-model assets, and normalizes FETV-EVAL result directories or CSV dumps. Human ratings cannot be recomputed. BLIP / FID / FVD need their packages and reference frames.

Dimensions

• Major content

  KindAxis

  What the clip depicts — the primary subject or event.

• Attribute control

  KindAxis

  Whether requested attributes (color, count, style) are realized.

• Prompt complexity

  KindAxis

  Simple vs compositional prompt difficulty.

Data composition

• Size

  619 prompts categorized on three orthogonal axes

• Sources

  lyx97/FETV · github.com · github.com

• Modalities

  text · video

• License

  cc-by-4.0

• Notes

  CLIPScore is the bounded official component verified in tree. Human ratings and BLIP / FID / FVD need imported scores or extra assets.

Metrics

Metric

• static_quality

  DirectionHigher is better

  FETV manual static-quality score.

• temporal_quality

  DirectionHigher is better

  FETV manual temporal-quality score.

• overall_alignment

  DirectionHigher is better

  FETV overall text-video alignment score.

• fine_grained_alignment

  DirectionHigher is better

  FETV fine-grained prompt-aspect alignment score.

• clip_score

  DirectionHigher is better

  FETV automatic CLIPScore component.

• blip_score

  DirectionHigher is better

  FETV automatic BLIPScore component.

• fid

  DirectionLower is better

  FETV Fréchet Inception Distance component.

• fvd

  DirectionLower is better

  FETV Fréchet Video Distance component.

• fetv_average

  DirectionHigher is better

  Mean emitted only when every declared FETV component score is available.

  PrimaryFETV Average

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Official paper Figure 7: static quality, temporal quality, and overall alignment (1–5 human). Rows follow the overall-alignment panel; ranks are not a single Overall. FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation (2023).

Model	Static quality	Temporal quality	Overall alignment
ModelScopeT2V	3.78	4.07	3.79
ZeroScope	3.52	3.72	3.57
Text2Video-Zero	4.11	2.04	3.42
CogVideo	3.24	3.30	3.10

How to run

Import a FETV-EVAL directory or official CSV:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id fetv \
  --mode official-validation \
  --official-results-path <official_results.csv-or-FETV-EVAL-dir> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/fetv/official-validation \
  --json

Select automatic families with --fetv-metrics (clip_score, blip_score, fid, fvd). FID / FVD need --fetv-fid-real-video-dir / --fetv-fvd-real-video-dir. Default import path: WORLDFOUNDRY_FETV_RESULTS_PATH. Optional: WORLDFOUNDRY_FETV_RUNTIME_ROOT, WORLDFOUNDRY_FETV_ARTIFACT_SCORE_DIR. Shared stacks: video_quality_motion_stack, fetv_blip_retrieval_source, fetv_stylegan_v_fvd_source.

In-tree pointers:

worldfoundry/data/benchmarks/tasks/external/fetv.yaml
worldfoundry/evaluation/tasks/execution/runners/fetv/runtime/fetv_eval
worldfoundry/data/benchmarks/assets/fetv/fetv_data.json

Outputs include scorecard.json (primary fetv_average when complete), benchmark_contract.json, and raw_metric_table.jsonl.

← Benchmark Hub