Environments

Choose the dependency stack for each model family, benchmark runtime, and validation stage.

On this page

Introduction

An environment is the dependency and launch context expected by a model or benchmark: Python packages, CUDA/Torch bucket, optional system tools, checkpoint layout, device assumptions, and command shape.

The repository still uses runtime_profile as a metadata field because it points to a concrete environment record in worldfoundry/data/models/runtime/profiles/<model-id>.yaml. In reader-facing docs, this page calls the same concept an environment.

WorldFoundry uses conda as the single supported environment path. The documented workflow creates one named conda environment for the main framework and separate conda environments only when an upstream simulator or official runtime requires isolation.

  1. Start with bash scripts/setup/bootstrap_worldfoundry.sh for model inference.
  2. Use bash scripts/setup/model_env_install.sh --list to inspect every model-to-env mapping, or --model <model-id> when you need the exact env for one model.
  3. Stay in the unified env when the resolved profile is worldfoundry-unified-<tier>.
  4. Switch to a dedicated env only when runtime/environments/<category>/<model-id>.yaml records an ABI or simulator constraint.
  5. Use benchmark-specific scripts only for official metrics, reference metrics, or physics stacks.

Python install surfaces

pyproject.toml exposes two intentionally different dependency surfaces:

  • Optional dependencies (extras) are the public, wheel-visible install surface for users and contributors. Install the smallest extra that matches the feature you need, for example python -m pip install -e ".[ui]".
  • Dependency groups are source-checkout tooling and resolver buckets. They are not wheel extras and are not selected with .[name]. With a PEP 735-capable pip, use a command such as python -m pip install --group lint; environment tooling may consume the same groups directly.

all is a curated general-runtime bundle, not the literal union of every extra. It deliberately excludes development tools, proprietary NVIDIA VFX bindings, training stacks, and model-specific ABI islands. In the catalog below, “In all” means that every package name declared by that extra is represented in all; install the named extra as well when its tighter version constraints are part of the runtime contract.

Extra

  • api

    Hosted model API clients and image/request helpers.

    In allYes

    Install.[api]

  • apple_pi

    Apple-PI evaluation runtime, judge clients, and media helpers.

    In allYes

    Install.[apple_pi]

  • hf

    Hugging Face and ModelScope asset download support.

    In allYes

    Install.[hf]

  • video

    Broad video, media, data, and 3D runtime dependencies.

    In allNo

    Install.[video]

  • vipe

    ViPE geometry/depth runtime and visualization stack.

    In allNo

    Install.[vipe]

  • molmoact2

    MolmoAct2 model serving stack with its narrow Transformers contract.

    In allNo

    Install.[molmoact2]

  • molmobot_pi0

    Pinned MolmoBot Pi0 policy runtime.

    In allNo

    Install.[molmobot_pi0]

  • metrics

    General evaluation metrics and dataset I/O.

    In allNo

    Install.[metrics]

  • distribution_metrics

    Perceptual and distribution metric models.

    In allNo

    Install.[distribution_metrics]

  • dvlt

    DVLT 3D/visualization runtime.

    In allNo

    Install.[dvlt]

  • ui

    Complete Studio/browser UI, including realtime transport dependencies.

    In allYes

    Install.[ui]

  • studio_realtime

    Minimal WebRTC/realtime Studio transport stack.

    In allYes

    Install.[studio_realtime]

  • rtx_postprocess

    Proprietary NVIDIA VFX post-processing; explicit opt-in only.

    In allNo

    Install.[rtx_postprocess]

  • tui

    Textual terminal UI.

    In allYes

    Install.[tui]

  • optimized_core

    Optional Torch-based core acceleration dependencies.

    In allYes

    Install.[optimized_core]

  • lora

    Load and merge released PEFT LoRA adapters for inference.

    In allNo

    Install.[lora]

  • profiling

    Model profiling helpers.

    In allNo

    Install.[profiling]

  • web

    Minimal FastAPI web service surface.

    In allYes

    Install.[web]

  • mcp

    Model Context Protocol server/client support.

    In allYes

    Install.[mcp]

  • worldscore

    WorldScore evaluation runtime.

    In allYes

    Install.[worldscore]

  • studio_pointcloud

    Point-cloud Studio visualization.

    In allYes

    Install.[studio_pointcloud]

  • studio_rerun

    Rerun-based Studio visualization.

    In allYes

    Install.[studio_rerun]

  • studio_geometry

    Geometry and Open3D Studio visualization.

    In allYes

    Install.[studio_geometry]

  • studio_plot

    Plotting and color utilities for Studio.

    In allNo

    Install.[studio_plot]

  • dev

    Build, pre-commit, and repository-pinned lint tooling.

    In allNo

    Install.[dev]

  • test

    Public CPU inference and packaging tests; no Torch or model downloads required.

    In allNo

    Install.[test]

  • eval_core

    Optional CPU evaluation dependencies.

    In allNo

    Install.[eval_core]

  • all

    Curated general runtime bundle described above.

    Install.[all]

The source-only dependency groups are narrower resolver building blocks:

  • lint — Repository-pinned pre-commit and Ruff toolchain.
  • flash_attn_fa25 — Flash-attention bucket for FA 2.5-compatible environments.
  • flash_attn_fa28 — Flash-attention bucket for FA 2.8-compatible environments.
  • runtime_clip_bundle — CLIP-oriented runtime compatibility bundle.
  • runtime_timm_bundle — Timm/Torch runtime compatibility bundle.
  • three_d_core — Shared point-cloud and 3D geometry packages.
  • transformers_high — Current Transformers/Diffusers compatibility bucket.
  • transformers_low — Legacy or model-constrained Transformers compatibility bucket.

These groups select Python requirements only. CUDA/Torch wheel selection, native extension builds, and isolated model environments still go through the setup scripts and runtime profiles documented below.

Clone-to-run setup scripts

Run these scripts from the repository root. They are part of the public release surface, not local machine notes.

  • scripts/setup/bootstrap_worldfoundry.sh — First command after cloning the repo. Creates/verifies the unified env, writes tmp/worldfoundry_unified_env.sh, optionally installs model-specific envs, optionally prepares model assets, and prints the Workspace launch command.
  • scripts/setup/unified_install.sh — Main GPU runtime only. Creates worldfoundry-unified-<cuda-tier> and installs requirements/worldfoundry-unified.txt, PyTorch CUDA wheels, optional flash-attn, and import/CUDA checks.
  • scripts/setup/model_env_install.sh — Per-model environment resolution. Reads worldfoundry/data/models/runtime/environments/**; routes compatible profiles back to the unified env and creates dedicated envs only for hard ABI conflicts.
  • scripts/inference/prepare_model_infer.sh — Checkpoint/Hugging Face preparation per model. Defaults to a plan/check-local flow. Add --download only after reviewing license, gated access, and storage requirements.
  • scripts/workspace/run_workspace.sh — Start the browser Workspace from an activated runtime. Keeps PYTHONPATH=., WORLDFOUNDRY_WORKSPACE_MAX_JOBS, and WORLDFOUNDRY_CKPT_DIR explicit.

The unified installer selects a Torch range that exists on the resolved wheel index: cu121 uses Torch 2.4–2.5, cu124 uses 2.4–2.6, and cu128 uses 2.7 or newer within the repository upper bound. After the CUDA-index install, the exact Torch/Torchvision/Torchaudio versions constrain the remaining PyPI pass so indirect dependencies cannot replace the selected CUDA build. On hosts without nvidia-smi, bootstrap explicitly uses the cu128 wheel index while enabling --allow-no-cuda and skipping flash-attn; direct installer calls with --cuda auto instead fail fast and ask for an explicit tier. If another selected extra requires a newer Torch than that tier publishes, dependency resolution fails explicitly instead of falling back to a PyPI CPU/different CUDA wheel.

Recommended fresh-clone path:

git clone https://github.com/OpenEnvision/WorldFoundry.git
cd WorldFoundry

bash scripts/setup/bootstrap_worldfoundry.sh
source tmp/worldfoundry_unified_env.sh
conda activate "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}"

Environment prefixes are machine-local artifacts. If an existing checkout reports a missing python3.x, missing native module, or libssl/libcrypto error, recreate the environment or run the public verifier:

bash scripts/setup/bootstrap_worldfoundry.sh --verify-only

Review a model before downloading checkpoints:

bash scripts/setup/bootstrap_worldfoundry.sh \
  --with-model matrix-game-2 \
  --prepare-model matrix-game-2

Download public assets only when ready:

export HF_TOKEN=<your-token-if-needed>
bash scripts/setup/bootstrap_worldfoundry.sh \
  --with-model matrix-game-2 \
  --prepare-model matrix-game-2 \
  --download-model-assets

--download-model-assets forwards to each model's preparation script. It does not bypass Hugging Face gates; users must accept model terms upstream and provide HF_TOKEN or HUGGING_FACE_HUB_TOKEN themselves.

For a multi-model workstation, install the unified env once and then verify exceptions explicitly:

bash scripts/setup/bootstrap_worldfoundry.sh \
  --with-model lingbot-world \
  --with-model octo \
  --prepare-model lingbot-world \
  --prepare-model octo

Start the Workspace after setup:

source tmp/worldfoundry_unified_env.sh
conda activate "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}"
PYTHONPATH=. WORLDFOUNDRY_WORKSPACE_MAX_JOBS=8 \
python -m worldfoundry.studio.serving.workspace --host 127.0.0.1 --port 7870

Use explicit roots when datasets or checkpoints should live on shared storage:

bash scripts/setup/bootstrap_worldfoundry.sh \
  --home /path/to/worldfoundry-home \
  --data-root /path/to/worldfoundry-data \
  --model-root /path/to/worldfoundry-models

Runtime variables

bootstrap_worldfoundry.sh and unified_install.sh write the same sourceable env file. Keep this file local to the machine; it contains paths but no secrets.

Variable

  • WORLDFOUNDRY_HOME

    Written by setup yes

    Root for local runtime state.

  • WORLDFOUNDRY_REPO_ROOT

    Written by setup yes

    Absolute path to the checked-out WorldFoundry repository.

  • WORLDFOUNDRY_CACHE_DIR

    Written by setup yes

    General cache root.

  • WORLDFOUNDRY_DATA_DIR

    Written by setup yes

    Dataset and benchmark data root.

  • WORLDFOUNDRY_MODEL_DIR

    Written by setup yes

    Model asset root.

  • WORLDFOUNDRY_CKPT_DIR

    Written by setup yes

    Compatibility checkpoint root for non-HF layouts and soft links.

  • WORLDFOUNDRY_ARTIFACT_DIR

    Written by setup yes

    Generated outputs and validation artifacts.

  • WORLDFOUNDRY_CONDA_ENVS_ROOT

    Written by setup yes

    Parent directory for unified and dedicated conda envs.

  • WORLDFOUNDRY_UNIFIED_ENV_PREFIX

    Written by setup yes

    Absolute path of the main conda env.

  • WORLDFOUNDRY_CUDA_PROFILE

    Written by setup yes

    Resolved CUDA wheel tier, normally cu128 on CUDA 12.8+ hosts.

  • HF_HOME, HF_HUB_CACHE, HF_DATASETS_CACHE

    Written by setup yes

    Hugging Face cache roots when using a WorldFoundry-managed cache.

  • HF_TOKEN, HUGGING_FACE_HUB_TOKEN

    Written by setup no

    Optional user-provided credentials for gated assets. Never commit these.

Prefer native Hugging Face loading (from_pretrained, snapshot_download, and standard HF cache variables) for public checkpoints. Use WORLDFOUNDRY_CKPT_DIR for upstream runtimes that require a fixed directory layout or for local soft links that preserve official paths without duplicating large files.

Benchmark dataset layouts

worldfoundry-eval dataset locate <org/name> resolves an existing dataset without network access. It checks the per-dataset environment override first, then local asset/data manifests, a direct data root, and finally Hugging Face cache snapshots. Direct roots accept org/name, org--name, and nested hfd_datasets layouts; hub caches use the standard datasets--org--name/snapshots/<revision> layout. Use worldfoundry-eval dataset plan <org/name> [...] to print deduplicated download commands without executing them.

Compiler caches

worldfoundry.core.execution.compile_cache configures persistent torch.compile caches for Inductor, Triton, the CUDA driver JIT, and torch C++/CUDA extensions. The compatibility import worldfoundry.runtime.compile_cache exposes the same API. The root is WORLDFOUNDRY_COMPILE_CACHE_DIR when set, otherwise ${WORLDFOUNDRY_CACHE_DIR}/compile. Generated-code caches are partitioned by a Torch/Triton/cuDNN/Python/accelerator fingerprint. Explicit TORCHINDUCTOR_CACHE_DIR, TRITON_CACHE_DIR, TORCH_EXTENSIONS_DIR, and CUDA_CACHE_PATH values take precedence when writable; an unwritable cache falls back to a process-local temporary directory with a warning.

On a multi-node cluster, use node-local storage rather than a shared NFS or project mount. The fingerprint separates incompatible software and accelerator stacks, but it does not isolate concurrent writers on different nodes:

export WORLDFOUNDRY_COMPILE_CACHE_DIR=/local-ssd/${USER}/worldfoundry-compile

Autotune defaults are applied with setdefault, so values exported before launch win. WorldFoundry defaults to TORCHINDUCTOR_AUTOTUNE_LOCAL_CACHE=1 and TORCHINDUCTOR_AUTOTUNE_REMOTE_CACHE=0; opt in to a remote autotune cache only when the cluster provides a compatible service.

Logging

The CLI flags and environment variables below feed the same opt-in logging pipeline, worldfoundry.core.observability.logging_setup.configure_logging. Explicit CLI flags take precedence over inherited environment values.

Setting

  • Level

    Console and file threshold; the CLI accepts DEBUG, INFO, WARNING, ERROR, or CRITICAL.

    CLI--log-level

    EnvironmentWORLDFOUNDRY_LOG_LEVEL

  • File sink

    Optional rotating file path. Output-owning CLI commands create a per-run events.jsonl when neither is set.

    CLI--log-file

    EnvironmentWORLDFOUNDRY_LOG_FILE

  • JSON Lines

    Makes the file sink JSON Lines; console output remains text.

    CLI--log-json

    EnvironmentWORLDFOUNDRY_LOG_JSON

  • Child context

    JSON object containing structured correlation fields such as run_id, job_id, benchmark_id, and model_id.

    EnvironmentWORLDFOUNDRY_LOG_CONTEXT

Framework-owned workers receive structured context through log_context_environment(). WORLDFOUNDRY_RUN_ID is also exported by output-owning CLI operations for operator visibility, but it is not a replacement for WORLDFOUNDRY_LOG_CONTEXT.

Structured log messages and fields redact sensitive key names, bearer tokens, common OpenAI/Hugging Face/GitHub/Slack/AWS token formats, and compact JWTs. Raw stdout/stderr capture remains fidelity-preserving and may use a separate line or argument scrubber; do not assume every captured byte passed through the structured logger. Loguru keeps native time/count retention semantics, while the stdlib fallback interprets retention as a rotated backup-file count.

Dedicated model environment exceptions

Start from worldfoundry-unified-<cuda-tier> for every model. Create a dedicated environment only for the profiles below, where the upstream runtime has a real ABI or framework-family conflict with the unified CUDA stack.

Install one exception with:

bash scripts/setup/model_env_install.sh --model <model-id>

For model setup, prefer scripts/inference/prepare_model_infer.sh; it installs the resolved environment first and then checks or downloads assets. Use scripts/setup/model_env_install.sh directly only when you want to prepare the conda env without touching checkpoints.

19 dedicated environmentsClick a row for the setup command and notes.

Visual & world12

ac3dac3d
Setup
bash scripts/inference/prepare_model_infer.sh ac3d --download

NotesOfficial runtime pins Python 3.10, PyTorch 2.4, and CUDA 12.4. Runtime code is in tree under worldfoundry/synthesis/visual_generation/ac3d/ac3d_runtime; only checkpoints/assets are external.

cosmos3worldfoundry-cosmos3-cu128
Setup
bash scripts/inference/prepare_model_infer.sh cosmos3 --download

NotesCosmos3 uses the native diffusion runner, checkpoint-shaped transformer, Wan VAE38, AVAE decoder, and core runtime policies. Guardrails remain an optional orchestration layer; Diffusers is not an inference backend.

gen3cgen3c
Setup
bash scripts/inference/prepare_model_infer.sh gen3c --download

NotesInference-only Cosmos Predict1 stack with Python 3.10, CUDA 12.4, torch 2.6, Transformer Engine, and MoGe. Setup applies the Transformer Engine link patch from released packages; it does not clone or build the training-only Apex repository.

lyra-1lyra
Setup
bash scripts/inference/prepare_model_infer.sh lyra-1 --download

NotesUses the same inference-only Cosmos Predict1 / Transformer Engine family as GEN3C.

magi-1magi-official-cu124
Setup
bash scripts/inference/prepare_model_infer.sh magi-1 --download

NotesOfficial stack is Python 3.10.12 with PyTorch 2.4 / CUDA 12.4. Keep flash-attn and flashinfer in this env because their wheels are ABI-sensitive.

warp-as-historywarp-as-history
Setup
bash scripts/inference/prepare_model_infer.sh warp-as-history --download

NotesPins Python 3.10, torch 2.5.0+cu121, diffusers 0.36, transformers 4.51, and peft 0.17.

hunyuanvideo-t2vhunyuanvideo-i2vhunyuanvideo-1.5-t2vhunyuanvideo-1.5-i2vworldfoundry-unified-cu128
Setup
bash scripts/inference/prepare_model_infer.sh hunyuanvideo-t2v --download

NotesAll four IDs use the shared native diffusion runner and role-based Hunyuan components. The former Diffusers/xFuser CLI environment is no longer an inference backend.

kairos-sensenovakairos-sensenova
Setup
bash scripts/inference/prepare_model_infer.sh kairos-sensenova --download

NotesUses the official torch/CUDA 12.6-class stack and native attention packages such as SageAttention. Use this env when unified CUDA does not provide the Kairos attention stack.

lingbot-worldlingbot-world-actlingbot-world
Setup
bash scripts/inference/prepare_model_infer.sh lingbot-world --download

NotesBase-Cam and the independently cataloged public Base-Act checkpoint share the in-tree runtime. Parity requires exact torch and flash-attn pins; the unified env produced invalid near-black act2cam output in upstream-style tests.

wonderworldwonderworld
Setup
bash scripts/inference/prepare_model_infer.sh wonderworld --download

NotesInference imports PyTorch3D CUDA extensions. Build or install PyTorch3D wheels compatible with this env before running the official path.

wonderjourneywonderworld
Setup
bash scripts/inference/prepare_model_infer.sh wonderjourney --download

NotesShares the WonderWorld PyTorch3D extension environment and uses in-tree WonderJourney runtime code.

hunyuan-game-craftHYGameCraft
Setup
bash scripts/inference/prepare_model_infer.sh hunyuan-game-craft --download

NotesOfficial inference relies on HYGameCraft plus torchrun sequence parallelism. The dedicated env includes the CUDA 12.4 toolkit and builds the pinned flash-attn 2.6.3 PyPI release without cloning its Git repository. Set cuda_visible_devices / torchrun_nproc_per_node explicitly on shared GPU machines.

Embodied & JAX4

octoworldfoundry-octo-jax
Setup
bash scripts/inference/prepare_model_infer.sh octo --download

NotesJAX CUDA 11.8, Flax 0.7.5, TensorFlow, dlimp, and Octo tokenizer assets are isolated from the PyTorch CUDA env.

lapaworldfoundry-lapa-jax-cu118
Setup
bash scripts/inference/prepare_model_infer.sh lapa --download

NotesUses the official Python 3.10, JAX 0.4.25 CUDA 11/cuDNN 8.6, Flax 0.8.2, Tux 0.0.2, and Transformers 4.40 inference stack. It stays isolated because Transformers 5 removed the Flax generation modules required by LAPA.

openvla-oftworldfoundry-vla-openvla-compat-cu121
Setup
bash scripts/inference/prepare_model_infer.sh openvla-oft --download

NotesPrismatic/OpenVLA-compatible stack with older transformers/timm/tokenizers. Use this env for LIBERO/ALOHA OpenVLA-OFT inference.

cogactworldfoundry-vla-openvla-compat-cu121
Setup
bash scripts/inference/prepare_model_infer.sh cogact --download

NotesShares the OpenVLA compatibility env and reuses a staged OpenVLA Llama tokenizer for inference-only loading. A token is only needed when no compatible local tokenizer or complete Llama-2 mirror exists.

Prepare-only3

lingbot-valingbot-va
Setup
bash scripts/inference/prepare_model_infer.sh lingbot-va --download

NotesPrepare-only profile for the official Python 3.10, torch 2.9, CUDA 12.6, flash-attn stack; stage the LingBot-VA base/posttrain checkpoints before rollout.

being-h05beingh
Setup
bash scripts/inference/prepare_model_infer.sh being-h05 --download

NotesPrepare-only profile. Stage all Being-H0.5 checkpoints; full get_action parity needs the upstream policy stack and more headroom than a single 80GB GPU profile.

dreamdojodreamdojo
Setup
bash scripts/inference/prepare_model_infer.sh dreamdojo --download

NotesPrepare-only profile for the official torchcodec/CUDA 12.8 setup on H100-class hosts. Stage the GR-1 eval dataset before enabling real command templates.

EvalCrafter is the only benchmark with a WorldFoundry-managed dedicated metric profile. Other benchmark-specific native stacks are listed below and are not claimed as configured until their own preflight passes.

Unified environment with extra runtime requirements

The models below should still use worldfoundry-unified-<cuda-tier>, but their docs should call out extra local assets, compiled wheels, or external services. These are not reasons to clone an external repository; the code path is in-tree.

  • Unified-env WRBench camera/world integrations (EasyAnimate, Hunyuan WorldPlay, HunyuanWorld-Voyager, InSpatio-World, Matrix-Game 2/3, ReCamMaster, SANA-WM, ATI, HyDRA, Spatia, minWM, MagicWorld, LiveWorld, VerseCrafter, VideoX-Fun camera variants, and the HY-World renderer) — Extra requirement: Hugging Face checkpoints declared by each profile; Spatia's default MapAnything checkpoint is non-commercial research only. Notes: These resolve to worldfoundry-unified-<cuda-tier>. Every model keeps an independent in-tree inference binding. Reusable Wan2.1, MapAnything, UniCeption, Uni3C, Depth Pro, STream3R, SAM3 video segmentation, Qwen3-VL entity extraction, and point-cloud projection components live under base_models; runtime config lives under worldfoundry/data. Runtime setup never clones source repositories. AC3D, Cosmos3, GEN3C, Hunyuan GameCraft, and both LingBot routes use the dedicated environments listed above.
  • SkyReels V3 — Extra requirement: CUDA compiler, flash-attn, and torchao for the official 8-GPU USP path. Notes: Use unified_install.sh without --skip-flash-attn or build flash-attn through scripts/setup/install_flash_attn.sh.
  • Matrix-Game-3 — Extra requirement: Flash-attn. Notes: FA3 may be requested by config; FA2 fallback is acceptable on A100 validation hosts.
  • HunyuanWorld-Mirror — Extra requirement: CUDA/PyTorch-matched gsplat wheel. Notes: Remains a unified-env model when the wheel matches the active torch stack.
  • pixelSplat — Extra requirement: Local Gaussian rasterization extension and re10k.ckpt. Notes: The model wrapper is in tree; only the extension/checkpoint are local prerequisites.
  • LingBot-Map — Extra requirement: Optional FlashInfer. Notes: The wrapper defaults to SDPA, so FlashInfer is not mandatory for baseline inference.
  • OpenPI / MME-VLA — Extra requirement: JAX policy dependencies, gsutil for some OpenPI checkpoint shards, and optional OpenPI-style websocket or robot-simulator services for full policy rollout. Notes: WorldFoundry keeps code in tree; these requirements are runtime services or checkpoint-transfer tools, not external repo clones. LAPA uses the isolated environment listed above.
  • Solaris — Extra requirement: JAX/Orbax checkpoint loading and official HF evaluation fixtures. Notes: Uses the unified env in WorldFoundry; document the dataset/checkpoint layout rather than a separate conda env.

Benchmark metric environments

Most benchmark official validation runs and official-result normalizers run in the unified environment. Full official metric suites can require a separate conda profile when their dependency stack would disturb model inference. Install those profiles through the same public setup entrypoint:

The complete per-benchmark mapping for all 42 Video Generation and World Model entries is in the benchmark runtime environment matrix. That matrix also records external API services, native extensions, checkpoint stacks, and the currently verified scope; needs_new_env: false alone is not a full-suite readiness claim.

bash scripts/setup/model_env_install.sh --model evalcrafter

EvalCrafter's supported three-metric CLIP path and final_result.txt import run in worldfoundry-unified-<tier>. The separate worldfoundry-evalcrafter-metrics profile mirrors the official revision's Python 3.9.17, pip 23.2.1, Torch 2.0.1, torchvision 0.15.2, TensorFlow 2.12.0, PaddlePaddle-GPU 2.5.1, MMCV 2.0.1, and MMAction2 1.1.0 package pins. It exists to prepare the upstream dependency surface, not to imply that WorldFoundry orchestrates the full scorer.

  • EvalCrafter — Environment boundary: Unified env for the three official CLIP-derived metrics and result import; official legacy environment for preparing the external full stack. Verified scope: The bounded metric path is verified. Environment preparation is documented, but complete 17-metric recomputation and leaderboard output remain unsupported.
  • iWorld-Bench — Environment boundary: Unified env, including the pinned in-tree ViPE native sources and CUDA build. WORLDFOUNDRY_IWORLD_BENCH_PYTHON is only an optional override for a non-standard ABI-compatible Python. Verified scope: One real 16-frame video passed the official memory_ability component path. The bundled three-prompt coverage and all nine component metrics were not run, so full-suite and leaderboard validity remain false.
  • VBench / VBench-2.0 / VBench++ — Environment boundary: Unified env plus dimension-specific checkpoints and optional perception packages. Verified scope: Bounded official dimensions only; each selected dimension must pass preflight.
  • T2V-SafetyBench — Environment boundary: Unified client environment plus a user-provided judge API endpoint/key. Verified scope: API failure is fail-closed; no judge API result is claimed from local fixture tests.

A separate environment is never inferred from a successful import. The catalog and runtime profile must name its Python executable or setup command, assets, and verified scope. If no reproducible environment recipe exists, the affected metric remains blocked rather than being reported through a fallback score.

The in-tree code keeps reusable foundation models under worldfoundry/base_models. These components are available for future full-suite work; their presence does not mean the bounded EvalCrafter runner currently executes them:

  • DeepFace runtime (worldfoundry/base_models/perception_core/face/deepface) — Available for future celebrity identity integration; the supported CLIP subset does not execute it.
  • DOVER runtime (worldfoundry/base_models/perception_core/video_quality/dover) — Available for future video-quality integration; the supported CLIP subset does not execute it.
  • VideoMAE action recipe (worldfoundry/base_models/perception_core/action_recognition/videomae_mmaction) — Only the EvalCrafter-used VideoMAEv2 config and Kinetics-400 label map are vendored. MMAction2 is installed as an environment dependency, not copied as framework source.

For the supported subset, stage CLIP ViT-B/32 under the standard checkpoint root or set its explicit override:

export WORLDFOUNDRY_CLIP_VIT_B32_MODEL_DIR="${WORLDFOUNDRY_HFD_ROOT}/openai--clip-vit-base-patch32"

See the EvalCrafter benchmark page for the no-proxy download and smoke command. The complete upstream suite also expects direct 0000.mp4 through 0699.mp4 files, but WorldFoundry does not currently claim that full path.

WorldFoundry container

The supported image is the repository-root docker/Dockerfile, built through docker/build_with_docker.sh. Dockerfiles under worldfoundry/base_models/perception_core/detection/yolo_world/mmyolo/docker/ (and the MMYOLO CircleCI Dockerfile) are upstream snapshot artifacts. WorldFoundry does not build or publish those images; they keep PyTorch 1.8/1.9-era CUDA bases and are not maintained against the current runtime. If you need an MMYOLO-specific image, build it from current upstream rather than this vendored copy.

Embodied simulator environments

The host setup is Docker plus the unified WorldFoundry environment:

bash scripts/setup/bootstrap_worldfoundry.sh
source tmp/worldfoundry_unified_env.sh
docker pull ghcr.io/allenai/vla-evaluation-harness/robotwin:latest
worldfoundry-eval embodied run \
  --config worldfoundry/data/benchmarks/eval_configs/embodied/robotwin/eval.yaml \
  --pull-docker

Benchmark configs that have a docker.image entry, either directly or through worldfoundry/data/benchmarks/runtime_profiles/official/<benchmark>.yaml, are launched through worldfoundry/evaluation/tasks/embodied/docker_runner.py. Inside the container, simulator dependencies are provided by the image. On the host, only mount datasets/checkpoints/results and provide the policy server endpoint.

Troubleshooting

When working with complex setups involving CUDA, simulators, or external evaluators, you may encounter environment issues:

1. CUDA mismatch

Symptom: CUDA error: no kernel image is available for execution on the device or PyTorch cannot find a GPU. Solution: Ensure your NVIDIA driver is up-to-date and supports the CUDA version installed by Conda. If you are on an older driver, you can force the installation script to use an older CUDA wheel bucket:

bash scripts/setup/unified_install.sh --cuda cu121

2. Missing system dependencies (e.g., for OpenGL/3D)

Symptom: Errors relating to libGL.so.1, EGL, or headless rendering failures when running embodied simulators or 3D benchmarks. Solution: Install the required system-level graphics libraries. On Ubuntu/Debian:

sudo apt-get update && sudo apt-get install -y libgl1-mesa-glx libglib2.0-0

3. Flash Attention installation fails

Symptom: The unified install script hangs or fails during flash-attn compilation. Solution: flash-attn is extremely sensitive to mismatched CUDA compiler (nvcc) versions. Ensure your nvcc --version matches the PyTorch CUDA version. Alternatively, you can disable flash-attn installation by passing --skip-flash-attn (if supported by your environment) or installing a pre-compiled wheel from the official release page.