---
title: VMBench
description: "Alibaba AMAP ICCV 2025 motion bench: 1,050 prompts, five local metrics. WorldFoundry vendors the scorer; it does not invent a board."
---

VMBench (AMAP-ML; ICCV 2025) evaluates **motion generation** with five perception-aligned local metrics — no hosted judge API:

- **PAS** — perceptible amplitude
- **OIS** — object integrity under motion
- **TCS** — temporal coherence
- **CAS** — commonsense adherence
- **MSS** — motion smoothness

Official suite: **1,050 motion-centric prompts**. Name videos `0001.mp4` … `1050.mp4`. `vmbench_average` is the mean of available PAS/OIS/TCS/CAS/MSS. License: Apache-2.0.

Paper: [VMBench (ICCV 2025)](https://openaccess.thecvf.com/content/ICCV2025/html/Ling_VMBench_A_Benchmark_for_Perception-Aligned_Video_Motion_Generation_ICCV_2025_paper.html). Project: [amap-ml.github.io/VMBench-Website](https://amap-ml.github.io/VMBench-Website/). Code: [AMAP-ML/VMBench](https://github.com/AMAP-ML/VMBench).

WorldFoundry **vendors the official metric code** and bundles the prompt suite (`worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json`). The official HF dataset endpoint was **not public** at integration time. Result / runtime wiring is present; **full CUDA metric recomputation is unverified**. Full recompute needs MMDetection / MMPose / DeepSpeed and detector / tracker checkpoints.

<BenchmarkPageSections id="vmbench" locale="en" />

## How to run

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id vmbench \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/vmbench/official-validation \
  --json
```

```bash
${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py \
  --run-official \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --prompt-manifest "${WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST}" \
  --output-dir tmp/vmbench/official-run \
  --json
```

Env: `WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST` (defaults to the in-tree suite), `WORLDFOUNDRY_VMBENCH_RESULTS_PATH`. Shared stack: `motion_stack`.

In-tree pointers: `worldfoundry/data/benchmarks/tasks/external/vmbench.yaml`, `worldfoundry/evaluation/tasks/execution/runners/vmbench/runtime/official`.

Outputs include `scorecard.json` (primary `vmbench_average`, leaderboard key `Avg`).

[← Benchmark Hub](/docs/evaluation/benchmark-hub)