Physics-IQ Verified

Last updated: Oct 7, 2026

On this page

Dimensions Data composition Metrics Leaderboard How to run

Physics-IQ Verified is the recommended revision of DeepMind's continuation protocol, packaged by Anates Labs. The task is the same as Physics-IQ Original: continue a filmed scene for 5 seconds. The dataset and score adapter are not. Verified ships repaired prompts/masks and Hugging Face media (Anates-Labs-Research/Physics-IQ-Verified), then scores per view before aggregation (physics_iq_verified_score = official final_score_view). Original uses GCS media and global final_score_orig. Do not mix the two.

Paper: Physics-IQ Verified. Project: physics-iq-verified.anates.ai. Shared engine provenance: google-deepmind/physics-IQ-benchmark.

WorldFoundry ships the scoring runtime in tree (run_physics_iq_official_runner.py --protocol verified) plus prompt metadata under worldfoundry/data/benchmarks/assets/physics-iq-verified. Official media (396 reference videos + 396 masks) still download from Hugging Face — they are not in the repo. One run covers 198 generated continuations; maintainers recommend averaging four runs for leaderboard claims.

Dimensions

Verified protocol
KindTrack
Anates Labs revision of Physics-IQ: verified prompts, verified masks, Hugging Face media, and per-view aggregation (final_score_view). Not interchangeable with Original.
Single-pass 198-video split
KindSplit
One run scores 198 generated 5-second continuations. Preflight expects 396 reference videos and 396 masks. Maintainers recommend averaging four runs for leaderboard claims.
Solid mechanics
KindCategory
Rigid contact, collisions, and object permanence in verified real scenes.
Fluids
KindCategory
Pours, splashes, and other fluid continuations.
Optics
KindCategory
Reflections, refractions, and lighting-dependent motion.
Thermodynamics
KindCategory
Melting, smoke, and heat-related change.
Magnetism
KindCategory
Magnetic attraction/repulsion in the verified set.

Data composition

Size
Same continuation task as Physics-IQ Original (5-second futures of filmed scenes), but a different dataset and score adapter. Official media live on Hugging Face Anates-Labs-Research/Physics-IQ-Verified (396 references + 396 masks). WorldFoundry tracks a 198-video single-pass contract and scores per view before aggregation.
Sources
Anates-Labs-Research/Physics-IQ-Verified · github.com
Modalities
image · video
Notes
Do not mix Original Google Cloud Storage media into --protocol verified. physics_iq_verified_mse is the official score_mse_view (higher is better after variance normalization), not Original's physics_iq_mse_penalty.

Metrics

Metric

physics_iq_verified_score
DirectionHigher is better
Primary verified aggregate (official final_score_view): arithmetic mean after per-scenario, per-view physical-variance normalization.
PrimaryPhysics-IQ Verified Score
physics_iq_verified_spatiotemporal
DirectionHigher is better
Per-view spatiotemporal IoU vs. the verified real continuation (where and when motion happens).
physics_iq_verified_spatial
DirectionHigher is better
Per-view spatial IoU: where action happens, regardless of timing.
physics_iq_verified_weighted_spatial
DirectionHigher is better
Per-view weighted spatial IoU: where and how much action happens.
physics_iq_verified_mse
DirectionHigher is better
Official score_mse_view after physical-variance normalization. Higher is better. This is not raw pixel MSE and not Original's physics_iq_mse_penalty.

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. physics-iq-verified.anates.ai.

physics-iq-verified.anates.ai · LiveOpen live board

How to run

Import an existing raw_metrics.csv:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id physics-iq-verified \
  --mode official-validation \
  --official-results-path <raw_metrics.csv> \
  --output-dir tmp/physics-iq-verified/official-validation \
  --json

Recompute from 198 generated continuations named by official scenario ids (--protocol is verified, not physics-iq-verified):

hf download Anates-Labs-Research/Physics-IQ-Verified \
  --repo-type dataset \
  --local-dir /path/to/physics-iq-verified

export WORLDFOUNDRY_PHYSICS_IQ_VERIFIED_ROOT=/path/to/physics-iq-verified
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/physics-iq-verified/generated_videos
${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/physics_iq/run_physics_iq_official_runner.py \
  --benchmark-id physics-iq-verified \
  --protocol verified \
  --run-official \
  --dataset-root "${WORLDFOUNDRY_PHYSICS_IQ_VERIFIED_ROOT}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/physics-iq-verified/official-run \
  --strict \
  --json

--strict fails on missing scenario videos. Outputs include scorecard.json (primary physics_iq_verified_score), raw_metrics.csv, and official_metrics.json. physics_iq_verified_mse is official score_mse_view (higher is better after variance normalization), not Original's physics_iq_mse_penalty.

← Benchmark Hub