iWorld-Bench

Last updated: Oct 7, 2026

On this page

Dimensions Data composition Metrics Leaderboard How to run

iWorld-Bench (Tsinghua, with Northeastern and SCUT; ICML 2026) evaluates interactive world models — generators that respond to external action sequences. A unified Action Generation Framework bridges text actions, camera matrices, and trajectories. The release filters 330k clips to 2.1k high-quality samples, then defines six task types / 4.9k test samples across visual generation, trajectory following, and memory (e.g. memory_symmetry for content that leaves and re-enters view).

iworldbench_average is the mean over available families. License: Apache-2.0.

Project: iworld-bench.com. Code: EmbodiedCity/iWorld-Bench. Data: EmbodiedCity/iWorld-Bench-Dataset.

WorldFoundry wraps the official metric runtime and normalizes report CSV/JSON. The catalog-bounded command uses --metric memory (pixel-only memory, no compiled ViPE). Full official GPU execution needs dataset metadata, VBench assets, and a CUDA-compatible ViPE build — the vendored ViPE tree lacks native extension sources.

Official paper teaser: six interactive tasks across UGV, UAV, human, and robot viewpoints.

Dimensions

Visual generation
KindAxis
Frame quality, brightness, color-temperature, sharpness, motion smoothness.
Trajectory following
KindAxis
trajectory_accuracy, trajectory_tolerance, trajectory_alignment against commanded actions/cameras.
Memory
KindAxis
memory_symmetry for content that leaves and re-enters view. Bounded verified metric: memory.

Data composition

Size
ICML 2026 interactive world-model bench. 330k clips filtered to 2.1k samples; six task types / 4.9k test samples over visual generation, trajectory following, and memory. Apache-2.0.
Sources
github.com
Modalities
video · action · camera
License
Apache-2.0
Notes
Bounded verified scope is one real 16-frame official memory_ability run. Full suite needs VBench assets and a CUDA ViPE build.

Metrics

Metric

image_quality
DirectionHigher is better
Frame-level image quality of generated videos.
brightness_consistency
DirectionHigher is better
Brightness stability across generated frames.
color_temperature_constraint
DirectionHigher is better
Color-temperature stability under the scene constraint.
sharpness_retention
DirectionHigher is better
Retention of sharpness through the rollout.
motion_smoothness
DirectionHigher is better
Temporal motion smoothness.
trajectory_accuracy
DirectionHigher is better
Accuracy of the executed trajectory against the commanded actions.
trajectory_tolerance
DirectionHigher is better
Tolerance-banded trajectory-following score.
memory_symmetry
DirectionHigher is better
Consistency of content that leaves and re-enters the view.
trajectory_alignment
DirectionHigher is better
Alignment between generated and commanded camera trajectories.
iworldbench_average
DirectionHigher is better
Mean over available iWorld-Bench metric families.
PrimaryiWorld-Bench Average

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. iworld-bench.com.

iworld-bench.com · LiveOpen live board

How to run

hf download EmbodiedCity/iWorld-Bench-Dataset --repo-type dataset \
  --revision 530c48cd3d8e9bb20ed685a96521c3a9b68cf6d2 \
  --local-dir "${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}"

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/iworldbench/run_iworldbench_official_runner.py \
  --run-official \
  --generated-videos-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --metric memory \
  --output-dir tmp/iworld-bench/memory \
  --json

--split selects task splits. --official-results-path imports an existing report. Env: WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT, WORLDFOUNDRY_IWORLD_BENCH_RESULTS_PATH, WORLDFOUNDRY_IWORLD_BENCH_RUNTIME_BACKEND. WORLDFOUNDRY_IWORLD_BENCH_PYTHON is an optional compatible-Python override, not a separate-env requirement.

In-tree pointer: worldfoundry/data/benchmarks/tasks/external/iworld-bench.yaml. Dataset layout: dataset/all_pack/metadata.csv, camera_following_metadata.csv, assets/.

Outputs include scorecard.json (primary iworldbench_average) and raw_metric_table.jsonl.

← Benchmark Hub