Model runtime

Runtime shape

A model call in this repository usually follows one chain: the pipeline is the public entrypoint, the operator prepares inputs, synthesis owns checkpoint or API execution, and representation or memory join only when the task needs geometry or multi-turn state. Every path converges on the same GenerationResult so Studio, the CLI, and benchmarks share one artifact contract.

Model-specific public wrappers should be thin, but inference implementations must not be repeated across model directories. Local diffusion families compose canonical components from worldfoundry/base_models/diffusion_model; only genuinely model-specific network blocks and adapters belong in a family package. See Native diffusion for ownership and dependency rules.

All model runtime integrations must be in-tree implementations. They must not depend on machine-local external checkouts. DiffSynth, Diffusers, LightX2V, and official model repositories are migration references rather than runtime backends. External repos and checkpoints remain provenance or acquisition metadata until the required inference code has been ported into canonical WorldFoundry components.

Use integration_status for the public support claim and backend_stage for implementation depth. Only integrated plus in_tree_runtime means a complete runnable integration. metadata_only, profile_only, and official_client_bridge_only are useful planning states, but they are not runnable integrations.

Current runtime contract

Every runnable method requires both a pipeline and an operator. The pipeline is the concrete class reachable from catalog pipeline_target or the canonical binding resolver. It owns model loading, normalized execution, and GenerationResult artifact semantics. The operator is the pipeline's input-side partner for validation, image/video loading, camera/action parsing, interaction shaping, and perception preprocessing. Operators may be thin, but they must exist.

Binding resolution should go through worldfoundry/evaluation/models/pipelines/bindings.py. Catalog loading, pipeline loading, runner resolution, and validation should call its helpers instead of re-parsing route strings independently. Runtime profiles may carry execution metadata, environment expectations, checkpoint paths, and command hints, but they do not replace a pipeline/operator pair. metadata_only or profile_only entries must not be advertised as runnable support.

worldfoundry/base_models/diffusion_model is the canonical local diffusion runtime and the only diffusion package boundary. Its root contracts plus role-based models/, recipes/, loaders/, runners/, schedulers/, optimizations/, and extensions/ form the architecture. The former diffsynth/, image/, video/, and catch-all common/ children have been removed; they are not supported ownership layers. Compatibility imports may point from a product module to a canonical component, but canonical components must never import back. Generic checkpoint materialization, state-dict loading, placement/offload, distributed execution, kernels, configuration, and IO come from worldfoundry/core; family packages cannot recreate them. worldfoundry/base_models/perception_core and worldfoundry/base_models/three_dimensions remain inference/runtime packages. Keep transforms and geometry helpers required for inference, but do not keep training datasets, dataloaders, callbacks, losses, augmentation-only modules, or experiment scripts there.

The built-in wan2.1-t2v-1.3b recipe is the first complete example of that boundary; other diffusion families (Matrix Game 3.5, Echo Memory, LTX, Cosmos Predict/Transfer, HunyuanVideo, Sana, FLUX.2) follow the same role assembly without owning their own pipeline, loader, scheduler, or runner. The per-family migration status — which roles each reuses, what was deleted, and the structural/tensor validation backing each recipe — is recorded in Native diffusion consolidation. Structural checkpoint verification is distinct from CUDA artifact parity, which remains a separate release gate per family.

Base abstractions

Four base classes define the assembly surface: PipelineABC (load/process/stream entrypoint), BaseOperator (input and interaction shaping only), BaseSynthesis (local native runtime or hosted API), and BaseMemory (prior state for streaming or multi-turn control). NativeDiffusionPipeline and NativeDiffusionRunner own local diffusion execution below the BaseSynthesis boundary. For the file-level table — which file defines each class and what each owns — see Runtime assembly; for the shared primitives they depend on, see Core infrastructure.

Directory roles

When deciding where a model fix belongs, ask whether the failure happens in input handling, inference, geometry postprocessing, or state management, then map it to the owning package: inputs → operators/, native inference and hosted APIs → synthesis/, 3D/4D inference and artifacts → base_models/three_dimensions/, streaming or multi-turn state → core/memory plus synthesis/*/memory. The directory-level layout of these packages, including the per-family synthesis.py files and where CameraCtrl/MotionCtrl live, is in Runtime assembly. Profile-only shells are not complete integrations.

Static inference presets live under worldfoundry/data/models/: catalog/ for metadata, bindings/ for pipeline entrypoints, runtime/profiles/ for execution requirements, runtime/environments/ for dependency specs, and runtime/configs/<model>/ for YAML/JSON presets and example inputs. Resolve a bundled file without depending on the working directory:

from worldfoundry.core.io.paths import package_data_path

config = package_data_path(
    "models", "runtime", "configs", "wonderworld", "inference.yaml"
)

Runtime manifests may use ${WORLDFOUNDRY_DATA_ROOT}/models/runtime/configs/<model>/<file>; WorldFoundry expands the token before launch. Keep relative inheritance inside each model directory. Configuration classes stay in Python modules. Training recipes, checkpoints, datasets, and generated job configs do not belong in packaged data.

Reference Implementation: OpenVLA

openvla is the first fully runnable VLA integration in this tree and the best example for understanding embodied-action runtime design. Its goal is simple: one checkpoint loader serves both Studio one-shot action traces and benchmark closed-loop rollouts, without requiring users to clone the upstream GitHub repository.

The embodied closed-loop path starts in runtime config. data/models/runtime/configs/vla_va_wam/openvla.yaml declares backend: callable_entrypoint and points policy_target to worldfoundry.synthesis.action_generation.openvla.runtime:predict_action. Embodied eval reads that config in evaluation/tasks/embodied/policy_adapter.py through build_policy_adapter(), then wraps the callable with CallableRuntimePolicyAdapter in evaluation/tasks/embodied/adapters/runtime_policy_adapters.py. Simulator rollouts therefore need only a language instruction and RGB observation to obtain one robot action, without going through the full pipeline lifecycle.

Studio and one-shot action traces still use the pipeline path. OpenVLAPipeline binds OpenVLASynthesis; OpenVLASynthesis.predict() resolves the runtime profile, selects a checkpoint, writes a plan JSON, and dispatches real inference unless plan_only is set. That path is better suited to single jobs that need artifact files, a run directory, and profile metadata rather than step-by-step simulator interaction.

Both entrypoints converge on synthesis/action_generation/openvla/openvla_runtime/inference.py. select_openvla_checkpoint() chooses a locally staged base or LIBERO fine-tuned checkpoint from unnorm_key. OpenVLARuntime then loads processor/model, builds the official OpenVLA prompt, runs predict_action, and writes normalized action-trace metadata. Prismatic/OpenVLA HF config, model, and processor code are vendored in openvla_runtime/configuration_prismatic.py, modeling_prismatic.py, and processing_prismatic.py, so the runtime does not depend on external repo checkouts or Hugging Face remote-code cache.

If you add a similar VLA policy, you usually need three pieces: a callable config under data/models/runtime/configs/vla_va_wam/, a predict_action() entrypoint in synthesis/action_generation/<model>/runtime.py, and an embodied adapter that can resolve image and instruction fields from simulator observations. Pipeline/synthesis layers can follow later for Studio and model-zoo one-shot runs, but they should not be prerequisites for closed-loop eval.

Pipeline authoring conventions

A typical pipeline file stores synthesis, operator, representation, and runtime config in its constructor; from_pretrained() stages checkpoints or initializes API clients; process() / __call__() converts user input into artifacts; and stream() is implemented only when Studio live controls or incremental output require it. Helpers should remain limited to reproducibility, credential handling, or artifact semantics. Do not carry training-script-style experiment logic into public pipelines.

Hosted API paths

Hosted provider paths such as Wan hosted family, Kling, Runway, Luma Ray, Hailuo hosted, Sora/Veo API, and World Labs live in separate wrappers under pipelines/, kept apart from local checkpoint runtimes. Hosted wrappers usually stage local images, build provider payloads, or wait for remote jobs to finish; they should not silently reuse local diffusion checkpoint loaders. Provider credentials belong in environment variables only. Do not place API keys in manifests, docs, tests, or generated reports.

Bridge to evaluation

The evaluation runner does not import pipeline files directly unless the catalog model runner resolves to that route.

resolve_pipeline_route() (evaluation/models/pipelines/bindings.py:359) and resolve_model_zoo_runner() (evaluation/models/runners/resolver.py:386) turn that chain into a runnable runner. The catalog-parsing, spec-merging, and runner-resolution mechanics behind them are documented in Evaluation Core — Model Resolution. Planned or profile-only entries may keep runtime_profile but should not expose a runner target; thin planning wrappers must not advertise pipeline_target as verified support.

Model family inventory

For directory-level guidance on where each kind of model fix belongs (pipelines/, operators/, synthesis/, base_models/three_dimensions/, core/memory), see Runtime assembly. For models with a claimed runnable runner, document public class behavior and public methods. For unclaimed or archived wrappers, keep only a short index entry instead of expanding implementation detail.