WorldFoundry
English中文

Physical AI Bench (PAI-Bench)

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to runIn-tree tracks and sidecars

PAI-Bench (SHI Labs / Georgia Tech, with CMU and NVIDIA Research; CVPR 2026) is a multi-track Physical AI suite: perception and control fidelity on generated video, not a physics-law prompt set like PhyGenBench and not a filmed continuation like Physics-IQ.

WorldFoundry's in-tree evaluator implements two tracks only:

  • PAI-Bench-G (--track generation): VBench / VBench++-style perception metrics plus VQA. Dataset: shi-labs/physical-ai-bench-generation.
  • PAI-Bench-C (--track conditional-generation): control fidelity (Canny, depth si-RMSE, segmentation, blur SSIM, LPIPS diversity). Dataset: shi-labs/physical-ai-bench-conditional-generation (~273 GB).

The paper may describe additional tracks. Those are not in this contract — do not pass --track understanding. The contract also does not mark a single primary metric; do not treat aesthetic_quality as the official total.

Paper: PAI-Bench. Code: SHI-Labs/physical-ai-bench.

Official paper teaser: PAI-Bench’s generation and understanding tracks for Physical AI.
Official paper teaser: PAI-Bench’s generation and understanding tracks for Physical AI.

Dimensions

  • PAI-Bench-G
    KindTrack

    Video generation track: VBench/VBench++-style perception metrics plus VQA. Needs --generated-video-dir and shi-labs/physical-ai-bench-generation.

  • PAI-Bench-C
    KindTrack

    Conditional generation track: control fidelity (Canny, depth, segmentation, blur, LPIPS diversity). Dataset is large (~273 GB).

Data composition

  • Size

    WorldFoundry's in-tree evaluator covers two tracks only: generation (shi-labs/physical-ai-bench-generation) and conditional-generation (shi-labs/physical-ai-bench-conditional-generation, ~273 GB). The paper may describe additional tracks; those are not in this contract. No official leaderboard is ingested.

  • Sources

    shi-labs/physical-ai-bench-generation · shi-labs/physical-ai-bench-conditional-generation · github.com

  • Splits

    generation · conditional-generation

  • Modalities

    text · image · video

  • Notes

    The contract does not mark a single primary metric. Generation and conditional-generation expose different families. Local scorecards are not SHI-Labs leaderboard submissions.

Metrics

Metric

  • aesthetic_quality
    DirectionHigher is better

    Generation-track VBench-style aesthetic quality.

  • background_consistency
    DirectionHigher is better

    Generation-track VBench-style background consistency.

  • imaging_quality
    DirectionHigher is better

    Generation-track VBench-style imaging quality.

  • motion_smoothness
    DirectionHigher is better

    Generation-track VBench-style motion smoothness.

  • overall_consistency
    DirectionHigher is better

    Generation-track VBench-style text–video consistency.

  • subject_consistency
    DirectionHigher is better

    Generation-track VBench-style subject identity consistency.

  • i2v_background
    DirectionHigher is better

    Generation-track VBench++ image-to-video background preservation.

  • i2v_subject
    DirectionHigher is better

    Generation-track VBench++ image-to-video subject preservation.

  • vqa_accuracy
    DirectionHigher is better

    Generation-track judge VQA on physical-plausibility questions.

  • dover_tech_score
    DirectionHigher is better

    Conditional-track DOVER technical quality.

  • blur_ssim
    DirectionHigher is better

    Conditional-track SSIM vs. blur conditioning.

  • canny_f1_score
    DirectionHigher is better

    Conditional-track F1 of Canny edges vs. the condition.

  • canny_precision
    DirectionHigher is better

    Conditional-track Canny precision vs. the condition.

  • canny_recall
    DirectionHigher is better

    Conditional-track Canny recall vs. the condition.

  • depth_si_rmse
    DirectionLower is better

    Conditional-track scale-invariant RMSE vs. conditioning depth. Lower is better.

  • seg_m_iou
    DirectionHigher is better

    Conditional-track mean IoU vs. conditioning segmentation.

  • seg_recall
    DirectionHigher is better

    Conditional-track recall of the conditioning segmentation.

  • lpips_diversity
    DirectionHigher is better

    Conditional-track LPIPS diversity across generations from the same condition.

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). PAI-Bench: A Comprehensive Benchmark For Physical AI (2025).

ModelOverall domain score
Veo3proprietary82.2
HunyuanVideo-I2V77.6
MAGI-1-4.5B77.5
LTX-Video-2B77.5
DynamicCrafter68.3

How to run

Score the generation track over staged data and generated videos:

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/physical_ai_bench/run_physical_ai_bench_official_runner.py \
  --run-official \
  --track generation \
  --dataset-root "${WORLDFOUNDRY_PHYSICAL_AI_BENCH_DATASET_ROOT}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/physical_ai_bench/generation \
  --json

Conditional track (checkpoints or prediction dirs as needed):

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  -m worldfoundry.evaluation.tasks.execution.runners.physical_ai_bench.run_physical_ai_bench_official_runner \
  --run-official \
  --track conditional-generation \
  --dataset-root "${WORLDFOUNDRY_PHYSICAL_AI_BENCH_DATASET_ROOT}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/physical_ai_bench/conditional \
  --json

--metrics restricts families for development. Generation VQA needs a prediction manifest or a configured Qwen-compatible judge (--judge-backend, --judge-model, --judge-base-url). Outputs include scorecard.json and official_results.json.

In-tree tracks and sidecars

This integration is inference-only. It never clones or imports the upstream repo or its third_party tree. Prompts, evaluator settings, and provenance live under worldfoundry/data/benchmarks/assets/physical-ai-bench (no SOURCE.json on purpose).

  • Generation reuses the checked-in VBench / VBench++ I2V runtimes. Example metric list: aesthetic_quality, background_consistency, imaging_quality, motion_smoothness, overall_consistency, subject_consistency. Binary physical VQA accepts a prediction manifest or a Qwen video judge.
  • Conditional-generation implements the published CPU transfer metrics locally and reuses shared DOVER, LPIPS, Video-Depth-Anything, GroundingDINO, and SAM2 base models. Precomputed sidecars: --pred-depth-dir (.npy) and --pred-segmentation-dir (JSON). Official dataset pickles load only with explicit --allow-trusted-pickle; prefer JSON segmentation sidecars.

Standard profile: worldfoundry-unified-cu128. Result normalization and the CPU Canny / blur / depth / segmentation metrics do not need a second env. Model-backed metrics need the matching WorldFoundry checkpoints.

PAI-Bench-G's Qwen2.5-VL-72B binary-VQA judge usually needs its own multi-GPU server. When it does not fit the unified process, start a vLLM OpenAI-compatible server and pass:

--judge-backend openai-compatible \
--judge-model Qwen/Qwen2.5-VL-72B-Instruct \
--judge-base-url http://localhost:8000/v1

That extra environment is only the optional model server. Benchmark code and aggregation stay in the unified WorldFoundry process.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets