WorldFoundry
English中文

Core infrastructure

The worldfoundry/core shared layer — loading, attention, distributed, VRAM, kernels, memory, and the rule that reusable primitives live here, not in family packages.

Edit this pageGitHubMarkdown

On this page
Why this layer existsSubpackage responsibilitiesDependency ruleWhere it is consumedWhat does not belong hereOwnership decisionsRegression gates

Why this layer exists

worldfoundry/core is the shared inference infrastructure that every model family reuses and none of them owns. Checkpoint loading, state-dict remapping, attention backends, distributed collectives, VRAM placement and offload, kernels, configuration, media IO, memory contracts, and safety guardrails all live here once. A family package (worldfoundry/base_models/*, worldfoundry/synthesis/*, worldfoundry/pipelines/*) composes these primitives; it must not recreate them.

This is the layer to read when you are about to copy a utility into a model directory. The rule is explicit: if multiple families actually reuse a capability, it belongs in core; an adapter tied to a specific model belongs with that model. See Native diffusion for how diffusion consumes this layer, and Model runtime for how pipelines and operators do.

Subpackage responsibilities

Subpackage

  • attention

    Backends, rotary embeddings, sequence layouts, KV caches, sparse and parallel attention.

    Start here backends/, rotary/, sequence/, cache/, sparse/, parallel/

  • distributed

    Process setup, collectives, model-parallel groups, sequence-parallel runtime and parameter sharding.

    Start here runtime/, collectives/, model_parallel/, sequence_parallel/, sharding/

  • model_loading

    Model construction, loading policy and weight conversion; checkpoints is the single owner of safe state-dict loading and assignment.

    Start here checkpoints/, factory.py, inference_model.py, lora.py

  • io

    Asset materialization, filesystems, structured formats and inference input datasets.

    Start here assets/, filesystem/, formats/, inputs/, paths.py, manifests.py

  • media

    Image, audio and video codecs, frame processing, output types and artifact previews.

    Start here codecs/, processing/, types.py, resolutions.py, artifacts.py

  • execution

    Device and process lifecycle, inference policy, runtime caches, CUDA graph replay and realtime orchestration.

    Start here graphs/, realtime/, inference.py, inference_runtime.py, process.py

  • configuration

    Structured/Python config, Hydra adapters, overrides, lazy instantiation and validation.

    Start here lazy_config/, hydra.py, hydra_utils.py, overrides.py, python.py, validators.py

  • nn

    Reusable neural blocks, transformer math, diffusion primitives, latent codecs and activation checkpointing.

    Start here blocks/, transformer/, diffusion/, latent/, checkpointing/

  • acceleration

    Reusable cache, token pruning and opt-in low-precision linear operators.

    Start here quantization/, cache.py, token_pruning.py, technology.py

  • kernels

    Operator implementations and dispatch for diffusion, MoE and quantized GEMM, including optional Triton providers.

    Start here capabilities.py, registry.py, diffusion.py, moe.py, quantized_gemm.py

  • vram

    Physical parameter placement, meta initialization, disk state, CPU offload and encoder lifetime.

    Start here device_map.py, initialization.py, disk_map.py, layerwise_offload.py, encoder_lifecycle.py

  • memory

    Semantic/context memory contracts, retrieval, conditioning and persistence.

    Start here base.py, conditioning.py, media.py, mosaic.py, retrieval.py, store.py

  • geometry

    Camera transforms, pose conditioning, trajectories and RGBD warping.

    Start here transforms.py, pose.py, trajectory.py, conditioning.py, warp.py

  • observability

    Logging, terminal formatting, timing, profiling and optional NVTX ranges.

    Start here logging_setup.py, formatting.py, colors.py, time.py, torchprofile.py

  • safety

    Safety filters and video input guardrails.

    Start here guardrails.py, video_io.py

  • utils

    General Python object/tree helpers and tensor/array utilities with no runtime or media ownership.

    Start here python/, tensors/

Dependency rule

core may depend on PyTorch, diffusers model/operator primitives, and narrow third-party libraries. It must not import worldfoundry.pipelines, worldfoundry.synthesis, worldfoundry.operators, worldfoundry.base_models family packages, or worldfoundry.evaluation, or worldfoundry.studio. The edge points out of family packages into core, never the reverse.

This keeps family packages thin: a model directory contains checkpoint-compatible tensor math and contract adapters, not a second copy of an offload manager, a distributed bootstrap, a kernel registry, or a quantization system. When you find yourself adding one of those to a family package, move it to core first and import it from there.

The diffusion package restates this rule for its own boundary in Native diffusion. Generic attention, transformer, loading, and offload primitives live in core; diffusion-domain contracts, schedulers, and recipes live in base_models/diffusion_model.

Where it is consumed

  • Diffusion — base_models/diffusion_model loads checkpoints through core/model_loading, runs attention through core/attention, and offloads through core/vram. Its optimizations/ re-exports core policy, not a parallel implementation.
  • Hosted and native runtimes — synthesis/ stages inputs through core/io and writes artifacts through core/media.
  • Memory — core/memory/base.py defines the contract; streaming and long-context state in synthesis/*/memory and action-policy trace state in synthesis/action_generation/memory.py implement it.
  • Distributed execution — core/distributed is the only place model-parallel groups are initialized; no family package boots its own NCCL group.

What does not belong here

Training entrypoints, training-only datasets and dataloaders, losses, callbacks, augmentation-only modules, and experiment scripts do not belong in core, nor in base_models. core is inference infrastructure; if a module only matters during training, it should not be added here.

For the directory-level map of every package, see Codebase map. For how a model call flows through these primitives, see Model runtime and Runtime assembly.

Ownership decisions

  • checkpoint is part of model_loading/checkpoints; weight loading, assignment and conversion have one owner.
  • Media codecs and the former video processing helpers share media; downloading and filesystem work stay in io.
  • CUDA graphs, prewarm, frame prefetch and overlap belong to execution; encoder placement/lifetime belongs to vram.
  • memory stays separate from vram: context/retrieval is a model input concern, while placement/offload manages physical memory.
  • Configuration helpers leave io, terminal formatting leaves io, and runtime scheduling leaves utils.
  • Wan sequence-parallel adapters and VGGT observation preprocessing live in their model families. Repository inventory audits live in scripts/model_zoo.

Public lazy function/class facades remain stable. Internal module imports use the current owners; removed directories do not contain compatibility wrappers.

Regression gates

tests/core/test_core_module_boundaries.py checks lazy targets and core imports without optional model dependencies. It also guards against dependencies from core into model/application packages. Runtime facade tests run in model environments.

Real-checkpoint changes also require make test-geometry: use fixed inputs, checkpoints, environments and seeds, compare numerical outputs against an accepted reference, and verify the replay hashes still match the current source. Moving files must preserve computation as well as importability.