AI2-THOR

AI2-THOR (Allen Institute for AI) is a Unity household simulator for embodied AI. The original iTHOR set is 120 artist-built rooms: 30 kitchens, 30 living rooms, 30 bedrooms, 30 bathrooms (FloorPlan1–30, 201–230, 301–330, 401–430). Agents take discrete actions (MoveAhead, RotateLeft, PickupObject, open/close, toggle) against physics and object state. The usual iTHOR split is 20 / 5 / 5 train / val / test rooms per type. Later packs (RoboTHOR, ProcTHOR) and language stacks (ALFRED, TEACh) sit on this engine — they are not the WorldFoundry default.

Paper: AI2-THOR: An Interactive 3D Environment for Visual AI. Project: ai2thor.allenai.org. Code: allenai/ai2thor (Apache-2.0).

WorldFoundry evaluates a native closed loop: worldfoundry-eval embodied run boots the Unity build and steps a policy that already emits simulator tokens. In-tree configs (FloorPlan1–FloorPlan3) are wiring checks, not ALFRED/TEACh. You can also import a rollout export via WORLDFOUNDRY_AI2THOR_RESULTS_PATH. This page does not run VLA inference — supply the policy or policy server yourself.

Dimensions

• iTHOR 120 rooms

  KindSuite

  Artist-built rooms: 30 kitchens, 30 living rooms, 30 bedrooms, 30 bathrooms (FloorPlan1–30, 201–230, 301–330, 401–430).

• Per-type train / val / test

  KindSplit

  Official iTHOR practice: first 20 rooms of each type for train, next 5 val, last 5 test.

• WorldFoundry wiring scenes

  KindSplit

  In-tree eval configs use FloorPlan1–3 only to prove Unity closed-loop wiring. Not an ALFRED or TEACh split.

• ALFRED / TEACh (not bundled)

  KindSuite

  Language-task stacks that sit on AI2-THOR. WorldFoundry does not ship those suites.

Data composition

• Size

  Unity household simulator from Ai2. The original iTHOR release is 120 interactive rooms with physics, object state changes, and discrete navigation/manipulation actions. Later scene packs (RoboTHOR, ProcTHOR) and language suites are separate downloads.

• Sources

  github.com

• Splits

  ai2thor_default

• Modalities

  language · vision · action

• License

  Apache-2.0

• Notes

  WorldFoundry boots Unity for FloorPlan wiring or imports a rollout export. It does not bundle ALFRED/TEACh assets or invent a public AI2-THOR model board.

Metrics

Metric

• success_rate

  DirectionHigher is better

  Primary episode success rate over evaluated rollouts.

  PrimarySuccess Rate

• task_success

  DirectionHigher is better

  Per-task success aggregate.

• episode_success

  DirectionHigher is better

  Per-episode success indicator aggregate.

• reward

  DirectionHigher is better

  Mean episode reward / return from the simulator.

Leaderboard

AI2-THOR is a Unity household simulator. The 2017 paper describes rooms, physics, and discrete actions; it does not publish a ranked policy table for the catalog task (ALFRED/TEACh are separate stacks). AI2-THOR: An Interactive 3D Environment for Visual AI (2017).

How to run

Closed-loop wiring (Unity package, simulator cache, and a CloudRendering or X11 display):

worldfoundry-eval embodied run \
  --config worldfoundry/data/benchmarks/eval_configs/embodied/ai2thor/floorplan1.yaml

Import an existing export:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id ai2thor \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --output-dir tmp/ai2thor/official-validation \
  --json

Optional WORLDFOUNDRY_AI2THOR_ROOT points at a local runtime/repo. Default import path: WORLDFOUNDRY_AI2THOR_RESULTS_PATH. Manifest: worldfoundry/data/benchmarks/tasks/external/ai2thor.yaml. Repo mirror: cache/vla_va_wam/repos/allenai--ai2thor. Outputs: scorecard.json (primary success_rate), run_manifest.json, results.jsonl, metrics/summary.json.

← Benchmark Hub