WorldFoundry
English中文

DEVIL Dynamics

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

DEVIL (UCAS and Baidu; NeurIPS 2024) evaluates text-to-video models on dynamics, not beauty. Prompts are stratified by dynamics grade; scores at several temporal granularities become three protocol metrics that the paper reports at >90% human correlation:

  • Dynamics range — how much content variation the model can produce.
  • Dynamics controllability — whether prompted dynamics levels are respected.
  • Dynamics quality — quality conditioned on the dynamics grade.

devil_dynamics_average is the mean of those three.

Paper: DEVIL. Code: MingXiangL/DEVIL.

WorldFoundry materializes metric rows from staged artifact scores (WORLDFOUNDRY_DEVIL_DYNAMICS_ARTIFACT_SCORE_DIR) or imports official JSON. Full recomputation still needs upstream checkpoints (RAFT, CLIP ViT-L/14 and ViT-B/32, ViCLIP, DINOv2, AMT, dynamics regression). The Gemini naturalness component is not part of local recomputation.

Dimensions

  • Dynamics range
    KindAxis

    How much content variation the model can produce across dynamics grades.

  • Dynamics controllability
    KindAxis

    Whether prompted dynamics levels are respected.

  • Dynamics quality
    KindAxis

    Quality conditioned on the official dynamics grade.

Data composition

  • Size

    NeurIPS 2024 dynamics-centric T2V protocol. Prompts are stratified by dynamics grade; three protocol metrics are reported at >90% human correlation.

  • Sources

    github.com

  • Modalities

    text · video

  • Notes

    WorldFoundry imports official JSON or staged artifact scores. Full recomputation needs RAFT/CLIP/ViCLIP/DINOv2/AMT plus the dynamics regression checkpoint. Gemini naturalness is not local.

Metrics

Metric

  • dynamics_range
    DirectionHigher is better

    Dynamics Range normalized benchmark score.

  • dynamics_controllability
    DirectionHigher is better

    Dynamics Controllability normalized benchmark score.

  • dynamics_quality
    DirectionHigher is better

    Dynamics Quality normalized benchmark score.

  • devil_dynamics_average
    DirectionHigher is better

    DEVIL Dynamics Average normalized benchmark score.

    PrimaryDEVIL Dynamics Average

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Paper dynamics-quality column. Evaluation of Text-to-Video Generation Models: A Dynamics Perspective (2024).

ModelD_quality
FreeNoise-Lavie66.3
OpenSora63.7
ModelScope62.6
StreamingT2V60.8
Show-157.7
Hotshot-XL52.2
Pika52.1
ZeroScope44.8

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id devil-dynamics \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/devil-dynamics/official-validation \
  --json

Generate one video per official DEVIL prompt; grade stratification comes from official prompt metadata. Default import: WORLDFOUNDRY_DEVIL_DYNAMICS_RESULTS_PATH. Runtime scores: WORLDFOUNDRY_DEVIL_DYNAMICS_ARTIFACT_SCORE_DIR.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/devil-dynamics.yaml, worldfoundry/evaluation/tasks/execution/runners/devil_dynamics.

Outputs include scorecard.json (primary devil_dynamics_average) and raw_metric_table.jsonl.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets