Embodied official runtime setup

WorldFoundry ships result normalizers, CLI entry points, and simulator interfaces under worldfoundry.evaluation.tasks.embodied. Checkpoints, simulator assets, Docker images, and rollout outputs are not in git. You point WorldFoundry at them with environment variables or CLI flags on your machine.

• Native closed-loop simulators (ai2thor, behavior1k, calvin, kinetix, libero, libero-mem, libero-plus, libero-pro, maniskill2, maniskill, mikasa, molmospaces, rlbench, robocasa, robocerebra, robomme, robotwin, simpler-env, vlabench) — Official result import, in-tree simulator API, Docker-backed rollout hooks
• Result import only (bridgedata-v2, metaworld) — Official-shaped result normalization

integration_status: integrated means the surface exists. It does not mean leaderboard evidence exists. Scorecards stay official_benchmark_verified=false until you complete the official protocol.

Quick start (no assets required)

After cloning, verify the catalog and runnable evaluation surfaces:

worldfoundry-eval zoo benchmarks --json

This confirms the bundled libero, robomme, and robotwin entries are discoverable. Actual scoring uses official rollout outputs or simulator execution.

Step 1: Set your roots

All paths resolve from WORLDFOUNDRY_* environment variables or CLI arguments. Replace placeholders with directories on your machine:

export WORLDFOUNDRY_DATA_DIR="$HOME/worldfoundry-data"
export WORLDFOUNDRY_HFD_DATASET_ROOT="$HOME/worldfoundry-data"   # HF dataset mirror root; may equal DATA_DIR
export WORLDFOUNDRY_CKPT_DIR="$HOME/worldfoundry-checkpoints"
export WORLDFOUNDRY_MODEL_DIR="$HOME/worldfoundry-models"
export WORLDFOUNDRY_CACHE_DIR="$HOME/worldfoundry-cache"
export WORLDFOUNDRY_ARTIFACT_DIR="$HOME/worldfoundry-artifacts"
export WORLDFOUNDRY_MODEL_SOURCE_DIR="$HOME/worldfoundry-upstream-repos"  # Root for cloned upstream repos

Layout is flexible. Common patterns:

• Flat HF mirror under WORLDFOUNDRY_HFD_DATASET_ROOT (lerobot__libero, TianxingChen__RoboTwin2.0, …)
• Custom subtree under WORLDFOUNDRY_DATA_DIR/datasets/...

As long as the matching WORLDFOUNDRY_*_ROOT / WORLDFOUNDRY_*_CHECKPOINT variables point at real files, WorldFoundry resolves them correctly.

Step 2: Generate scaffold, then fill in your paths

Generate templates and placeholder directories (does not download large files for you):

PYTHONPATH=. ${WORLDFOUNDRY_UNIFIED_PYTHON} scripts/setup/prepare_embodied_official_assets.py \
  --create-dirs \
  --json

Outputs (default under tmp/benchmark_zoo/embodied_official_assets/, override with --output-root):

• embodied_official_env.sh — Sourceable env template; your existing exports win (${VAR:-default})
• local_assets_manifest.yaml — Local asset manifest; copy outside git and set WORLDFOUNDRY_LOCAL_ASSET_MANIFEST
• check_repo_assets.sh — Checks configured source/runtime roots and reports missing paths
• download_public_assets.sh — Public HF download commands
• validate_official_results.sh — Batch import after rollouts

Load and override with your paths:

source tmp/benchmark_zoo/embodied_official_assets/embodied_official_env.sh
bash tmp/benchmark_zoo/embodied_official_assets/check_repo_assets.sh

export WORLDFOUNDRY_LIBERO_ROOT="$WORLDFOUNDRY_MODEL_SOURCE_DIR/LIBERO"
export WORLDFOUNDRY_LIBERO_POLICY_CHECKPOINT="$WORLDFOUNDRY_CKPT_DIR/your-policy-for-libero"
export WORLDFOUNDRY_LIBERO_RESULTS_PATH="$WORLDFOUNDRY_ARTIFACT_DIR/runs/libero/official_results.json"

Or copy worldfoundry/data/benchmarks/local_assets.example.yaml, edit ${WORLDFOUNDRY_*} placeholders, then:

export WORLDFOUNDRY_LOCAL_ASSET_MANIFEST=/path/to/your/local_assets_manifest.yaml

Step 3: Choose how far to validate

• Framework wiring — worldfoundry-eval zoo benchmarks --json. Needs GPU / assets: No
• Official result import — worldfoundry-eval zoo benchmark-run --benchmark-id robotwin --mode official-validation --official-results-path <official_results.json> --output-dir tmp/benchmark_zoo/official_validation/robotwin --json. Needs GPU / assets: No
• List missing env / paths — worldfoundry-eval zoo benchmark-show --benchmark-id robotwin --json. Needs GPU / assets: No (reports gaps)
• Import existing official results — See Import official results below. Needs GPU / assets: Result files only
• Full simulator rollout — Run upstream evaluator in cloned repo. Needs GPU / assets: Yes

Environment strategy

Start with the unified CUDA environment:

bash scripts/setup/bootstrap_worldfoundry.sh --cuda cu128
source tmp/worldfoundry_unified_env.sh

• LIBERO, CALVIN, RoboCasa, MiKASA, RoboCerebra, RoboMME — Preferred env: worldfoundry-unified-cu128. When to isolate: robosuite/MuJoCo pin conflicts
• LIBERO-Plus — Preferred env: libero-plus profile. When to isolate: May not coexist with base LIBERO
• BEHAVIOR-1K — Preferred env: Dedicated OmniGibson/Isaac Sim. When to isolate: Large assets + upstream terms
• Kinetix — Preferred env: Dedicated JAX env
• RLBench — Preferred env: Dedicated CoppeliaSim/PyRep
• RoboTwin, VLABench, ManiSkill2, SimplerEnv — Preferred env: Unified for import; external simulator runtime for rollouts. When to isolate: SAPIEN/Curobo etc.
• AI2-THOR — Preferred env: External simulator runtime. When to isolate: Unity binary + CloudRendering/X11
• Octo/OpenPI policies — Preferred env: Model-specific profile

List model env mappings: bash scripts/setup/model_env_install.sh --list

Benchmark environment variables

Each benchmark uses separate env vars for your checkout, checkpoint, and results:

Upstream code roots use WORLDFOUNDRY_<BENCHMARK>_ROOT. The full list is in the generated embodied_official_env.sh.

Validate and import

Import official results (when you already have rollout outputs):

worldfoundry-eval zoo benchmark-run \
  --benchmark-id libero \
  --mode official-validation \
  --official-results-path "$WORLDFOUNDRY_LIBERO_RESULTS_PATH" \
  --output-dir "$WORLDFOUNDRY_ARTIFACT_DIR/validation/libero" \
  --json

You can pass --official-results-path and --benchmark-data-root directly on the CLI without exporting env vars first.

Review scorecard.json, raw_results.jsonl, and evaluation/summary.json. official_benchmark_verified=false is expected for import-only runs.

Full rollout

Run the Docker-backed official evaluator or VLA evaluation harness with your policy checkpoint, write result files, then import with official-validation above.

Before rollout, record: benchmark ID, upstream commit, simulator versions, dataset split, checkpoint path, action/observation contract, and result file path.

For a real policy, replace model_id in the eval config with a configured VLA policy, or point server.url at an external policy server. The policy can return {"token": "forward"} or an equivalent discrete action payload.

Local validation inputs

Prepare local simulator inputs separately; fixture datasets are not distributed with the public repository. Validation runs do not constitute leaderboard evidence.

Maintainer notes

Release gates and official_benchmark_verified labeling are documented in Validation evidence.