VMBench

Last updated: Oct 7, 2026

On this page

Dimensions Data composition Metrics Leaderboard How to run

VMBench (AMAP-ML; ICCV 2025) evaluates motion generation with five perception-aligned local metrics — no hosted judge API:

PAS — perceptible amplitude
OIS — object integrity under motion
TCS — temporal coherence
CAS — commonsense adherence
MSS — motion smoothness

Official suite: 1,050 motion-centric prompts. Name videos 0001.mp4 … 1050.mp4. vmbench_average is the mean of available PAS/OIS/TCS/CAS/MSS. License: Apache-2.0.

Paper: VMBench (ICCV 2025). Project: amap-ml.github.io/VMBench-Website. Code: AMAP-ML/VMBench.

WorldFoundry vendors the official metric code and bundles the prompt suite (worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json). The official HF dataset endpoint was not public at integration time. Result / runtime wiring is present; full CUDA metric recomputation is unverified. Full recompute needs MMDetection / MMPose / DeepSpeed and detector / tracker checkpoints.

Official paper teaser: six motion-pattern categories in VMBench.

Dimensions

PAS
KindAxis
Perceptible amplitude.
OIS
KindAxis
Object integrity under motion.
TCS
KindAxis
Temporal coherence.
CAS
KindAxis
Commonsense adherence.
MSS
KindAxis
Motion smoothness.

Data composition

Size
Alibaba AMAP ICCV 2025 motion bench. 1,050 prompts named 0001.mp4–1050.mp4. Five local metrics, no hosted judge. Apache-2.0. Official HF dataset was not public at integration.
Sources
github.com
License
Apache-2.0
Notes
In-tree prompt bundle is the tracked source. Result/runtime wiring is present; full CUDA recomputation (MMDetection/MMPose/DeepSpeed) is unverified.

Metrics

Metric

perceptible_amplitude_score
DirectionHigher is better
Official VMBench PAS metric for perceptible motion amplitude.
object_integrity_score
DirectionHigher is better
Official VMBench OIS metric for object identity and integrity under motion.
temporal_coherence_score
DirectionHigher is better
Official VMBench TCS metric for temporal coherence.
commonsense_adherence_score
DirectionHigher is better
Official VMBench CAS metric for action and motion commonsense adherence.
motion_smoothness_score
DirectionHigher is better
Official VMBench MSS metric for motion smoothness.
vmbench_average
DirectionHigher is better
Mean of available official VMBench PAS/OIS/TCS/CAS/MSS component scores.
PrimaryVMBench Average

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). VMBench: A Benchmark for Perception-Aligned Video Motion Generation (2025).

Model	Avg (PMM)
Wan2.1	78.4
HunyuanVideo	63.4
CogVideoX-5B	60.6
OpenSora-Plan-v1.3.0	58.9
Mochi 1	53.2
OpenSora-v1.2	51.6

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id vmbench \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/vmbench/official-validation \
  --json

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py \
  --run-official \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --prompt-manifest "${WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST}" \
  --output-dir tmp/vmbench/official-run \
  --json

Env: WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST (defaults to the in-tree suite), WORLDFOUNDRY_VMBENCH_RESULTS_PATH. Shared stack: motion_stack.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/vmbench.yaml, worldfoundry/evaluation/tasks/execution/runners/vmbench/runtime/official.

Outputs include scorecard.json (primary vmbench_average, leaderboard key Avg).

← Benchmark Hub