---
title: Physical AI Bench (PAI-Bench)
description: "CVPR 2026 Physical AI suite. WorldFoundry scores two tracks in tree: generation (VBench-style + VQA) and conditional generation (control fidelity). No single primary metric."
---

PAI-Bench (SHI Labs / Georgia Tech, with CMU and NVIDIA Research; CVPR 2026) is a **multi-track Physical AI** suite: perception and control fidelity on generated video, not a physics-law prompt set like [PhyGenBench](/docs/evaluation/benchmark-hub/phygenbench) and not a filmed continuation like [Physics-IQ](/docs/evaluation/benchmark-hub/physics-iq).

WorldFoundry's in-tree evaluator implements **two tracks only**:

- **PAI-Bench-G** (`--track generation`): VBench / VBench++-style perception metrics plus VQA. Dataset: `shi-labs/physical-ai-bench-generation`.
- **PAI-Bench-C** (`--track conditional-generation`): control fidelity (Canny, depth si-RMSE, segmentation, blur SSIM, LPIPS diversity). Dataset: `shi-labs/physical-ai-bench-conditional-generation` (~273 GB).

The paper may describe additional tracks. Those are **not** in this contract — do not pass `--track understanding`. The contract also does **not** mark a single primary metric; do not treat `aesthetic_quality` as the official total.

Paper: [PAI-Bench](https://arxiv.org/abs/2512.01989). Code: [SHI-Labs/physical-ai-bench](https://github.com/SHI-Labs/physical-ai-bench).

<BenchmarkPageSections id="physical-ai-bench" locale="en" />

## How to run

Score the generation track over staged data and generated videos:

```bash
${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/physical_ai_bench/run_physical_ai_bench_official_runner.py \
  --run-official \
  --track generation \
  --dataset-root "${WORLDFOUNDRY_PHYSICAL_AI_BENCH_DATASET_ROOT}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/physical_ai_bench/generation \
  --json
```

Conditional track (checkpoints or prediction dirs as needed):

```bash
${WORLDFOUNDRY_UNIFIED_PYTHON} \
  -m worldfoundry.evaluation.tasks.execution.runners.physical_ai_bench.run_physical_ai_bench_official_runner \
  --run-official \
  --track conditional-generation \
  --dataset-root "${WORLDFOUNDRY_PHYSICAL_AI_BENCH_DATASET_ROOT}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/physical_ai_bench/conditional \
  --json
```

`--metrics` restricts families for development. Generation VQA needs a prediction manifest or a configured Qwen-compatible judge (`--judge-backend`, `--judge-model`, `--judge-base-url`). Outputs include `scorecard.json` and `official_results.json`.

## In-tree tracks and sidecars

This integration is inference-only. It never clones or imports the upstream repo or its `third_party` tree. Prompts, evaluator settings, and provenance live under `worldfoundry/data/benchmarks/assets/physical-ai-bench` (no `SOURCE.json` on purpose).

- **Generation** reuses the checked-in VBench / VBench++ I2V runtimes. Example metric list: `aesthetic_quality`, `background_consistency`, `imaging_quality`, `motion_smoothness`, `overall_consistency`, `subject_consistency`. Binary physical VQA accepts a prediction manifest **or** a Qwen video judge.
- **Conditional-generation** implements the published CPU transfer metrics locally and reuses shared DOVER, LPIPS, Video-Depth-Anything, GroundingDINO, and SAM2 base models. Precomputed sidecars: `--pred-depth-dir` (`.npy`) and `--pred-segmentation-dir` (JSON). Official dataset pickles load only with explicit `--allow-trusted-pickle`; prefer JSON segmentation sidecars.

Standard profile: `worldfoundry-unified-cu128`. Result normalization and the CPU Canny / blur / depth / segmentation metrics do not need a second env. Model-backed metrics need the matching WorldFoundry checkpoints.

PAI-Bench-G's Qwen2.5-VL-72B binary-VQA judge usually needs its own multi-GPU server. When it does not fit the unified process, start a vLLM OpenAI-compatible server and pass:

```bash
--judge-backend openai-compatible \
--judge-model Qwen/Qwen2.5-VL-72B-Instruct \
--judge-base-url http://localhost:8000/v1
```

That extra environment is only the optional model server. Benchmark code and aggregation stay in the unified WorldFoundry process.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)