ChronoMagic-Bench

ChronoMagic-Bench (PKU Yuan Group, with University of Rochester) asks whether a text-to-video model can synthesize metamorphic time-lapse: large physical, biological, or chemical state change (ice melting, flowers blooming, construction) while staying temporally coherent. The official close split has 1,649 prompts. Two core families:

• CHScore — tracking-point coherence via CoTracker.
• MTScore / GPT-4o-MTScore — metamorphic amplitude (InternVideo2 or GPT-4o).

The Hugging Face discovery table records 1,799 rows. Companion training sets ChronoMagic / Pro / ProH are listed upstream. Leaderboard parity additionally needs UMT-FVD and UMTScore, which WorldFoundry does not treat as complete.

Paper: ChronoMagic-Bench. Project: pku-yuangroup.github.io/ChronoMagic-Bench. Code: PKU-YuanGroup/ChronoMagic-Bench. Data: BestWishYsh/ChronoMagic-Bench (CC-BY-4.0).

WorldFoundry vendors the official runtime and runs selected component scripts (chscore, mtscore, gpt4o_mtscore) then official JSON normalization. CHScore is the bounded official component verified in tree. Primary key: chronomagic_score.

Dimensions

• Temporal coherence

  KindAxis

  CHScore tracking-point coherence via CoTracker.

• Metamorphic amplitude

  KindAxis

  MTScore / GPT-4o-MTScore for large physical or biological state change.

Data composition

• Size

  1,649 closed-set prompts; ChronoMagic-Bench dataset discovery records 1,799 rows

• Sources

  BestWishYsh/ChronoMagic-Bench · BestWishYsh/ChronoMagic · BestWishYsh/ChronoMagic-Pro · BestWishYsh/ChronoMagic-ProH · github.com

• Splits

  close

• Modalities

  text · video

• License

  cc-by-4.0

• Notes

  CHScore is the bounded official component verified in tree. Leaderboard parity still needs MTScore, GPT-4o-MTScore, UMT-FVD, UMTScore, full prompt coverage, and the official submission protocol.

Metrics

Metric

• chronomagic_score

  DirectionHigher is better

  Primary ChronoMagic family used on the scorecard. Official core components are CHScore (coherence) and MTScore / GPT-4o-MTScore (metamorphic amplitude).

  PrimaryChronoMagic Score

• temporal_transformation

  DirectionHigher is better

  Time-lapse transformation and ordering consistency, when the official dump exposes that family.

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. Hugging Face Space (BestWishYsh).

How to run

CHScore on generated close-split videos:

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/chronomagic_bench/run_chronomagic_bench_official_runner.py \
  --chronomagic-root worldfoundry/evaluation/tasks/execution/runners/chronomagic_bench/runtime/chronomagic_bench \
  --components chscore \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --model-name worldfoundry \
  --eval-type close \
  --model-pth-chscore "${WORLDFOUNDRY_CHRONOMAGIC_CHSCORE_CKPT}" \
  --output-dir tmp/chronomagic-bench/chscore \
  --python "${WORLDFOUNDRY_UNIFIED_PYTHON}" \
  --json

--components also accepts mtscore, gpt4o_mtscore, or all. MTScore needs WORLDFOUNDRY_CHRONOMAGIC_MTSCORE_CKPT; GPT-4o-MTScore needs OPENAI_API_KEY or WORLDFOUNDRY_CHRONOMAGIC_OPENAI_API. Existing dumps import via --official-results-path / WORLDFOUNDRY_CHRONOMAGIC_RESULTS_PATH. Dataset discovery root: WORLDFOUNDRY_CHRONOMAGIC_DATASET_ROOT.

hf download BestWishYsh/ChronoMagic-Bench --repo-type dataset \
  --local-dir cache/worldfoundry/data/hfd_datasets/BestWishYsh/ChronoMagic-Bench

In-tree pointer: worldfoundry/evaluation/tasks/execution/runners/chronomagic_bench/runtime/chronomagic_bench.

Outputs include scorecard.json (primary chronomagic_score), raw_metric_table.jsonl, and captured upstream logs when the scorer runs.

← Benchmark Hub