Add a model

Adding a model to WorldFoundry is not “drop a checkpoint into a folder and hope the CLI finds it.” The catalog records identity and provenance. A pipeline loads weights and runs inference. An operator turns a GenerationRequest into the inputs that pipeline understands. A binding tells the evaluation runner which class to import. A runtime profile describes the environment and assets. None of those pieces is optional if you want the model to be a first-class, rerunnable method in this repository.

This guide walks through that route in the order a reviewer will look at it. The companion pages are Models for existing IDs, Run inference for launching a demo, Local assets for checkpoint layout, and Design for why generation and scoring stay on opposite sides of an artifact boundary.

Keep the first PR narrow

The first pull request should prove one route end to end. Pick one model family, one operator, one pipeline, one runtime profile if the environment is not the unified default, one resolved runner path, and one focused run-evidence report. That is enough to show that catalog → binding → operator → pipeline → artifact works on a machine a reviewer can reconstruct.

Do not wait until every checkpoint variant, every task alias, and every benchmark cell is green. Those can follow once the spine exists. A wide first PR that also rewrites shared loaders, invents a second binding parser, or vendors a training tree is harder to review and easier to regress.

What an integration actually touches

A WorldFoundry model is a stack of five surfaces. The pipeline is the public entrypoint: a class under worldfoundry/pipelines/<family>/ that exposes from_pretrained() and a deterministic process() or __call__(). That class owns checkpoint construction and the native generate call. It should not parse prompts, decode camera trajectories, or decide which image tensor is “the first frame.” That work belongs to the operator.

The operator lives under worldfoundry/operators/. Every runnable method needs one, even when the first version is thin. Its job is to validate the request, load media, parse camera or action controls, and shape interaction state so the pipeline sees a stable, typed input. If you skip the operator and stuff that logic into the pipeline, the next benchmark or Studio form will have nowhere clean to attach.

Backend-specific inference code — the vendor inference.py, the transformer block, the tokenizer wrapper — belongs in worldfoundry/synthesis/, worldfoundry/base_models/, following the layout already used by that family. Those packages are runtime primitives. They are not a place for training loops, dataset builders, dataloaders, augmentation pipelines, or Lightning callbacks.

The evaluation binding is how CLI, TUI, and the generic benchmark runner find the method. Resolution always goes through catalog pipeline_target or pipeline_binding, then through worldfoundry/evaluation/models/pipelines/bindings.py. That file is the only place that should normalize route strings. Do not add a second parser in catalog loading, in the resolver, or in a check helper.

Finally, catalog metadata is the public record. The YAML under worldfoundry/data/models/catalog/<category>/<model-id>.yaml holds source, license, checkpoint refs, task coverage, runtime profile, blockers, and integration.status. The support matrix does not trust those fields as proof. It recomputes readiness from staged evidence.

The runnable route contract

WorldFoundry resolves every runnable method through the same public contract. A method is runnable only when a concrete pipeline class can be imported. You name that class with pipeline_target (module:Class) or with a pipeline_binding id that bindings.py already knows. If neither is present, worldfoundry-eval run <model-id> will fail before any GPU work starts, and that is intentional.

A runtime profile is not a substitute for that route. Profiles describe conda or Docker environment, checkpoint roles, launch commands, and execution metadata. They help a machine prepare to run. They do not, by themselves, count as “this model is integrated.” A catalog row that only lists a profile and no pipeline will stay planned or blocked.

base_models/perception_core and base_models/three_dimensions follow the same rule more strictly: inference and runtime helpers only. If the official tree ships a training script next to inference.py, leave the training side out of the in-tree package. Port the infer path; do not import the trainer.

Implement the route

Start with the pipeline. Put it under worldfoundry/pipelines/<family>/ so the family stays browsable. from_pretrained() should accept the same kinds of arguments the loader already knows how to pass — model path or options dict, device, model id — and should fail loudly when a required checkpoint is missing. process() / __call__() should return something the WorldFoundry runner can wrap as a GenerationResult: status, timings, errors, and durable artifact paths. Do not return an in-memory tensor as the only output. Evaluation and Studio need a file they can reopen after the process exits.

Add the operator in the same change set. Even a first-pass operator that only checks “prompt is a string” and “image path exists” is better than a pipeline that silently accepts a dict of unknown keys. As the model grows — multi-view, action chunks, camera rigs — keep that growth in the operator so the pipeline stays a checkpoint runner.

Place the vendor or in-house inference implementation according to the existing tree. Video synthesis usually lands under synthesis/visual_generation/. Interactive world models often sit next to a sibling that already vendors an official infer-only runtime. Perception and 3D helpers go under base_models/. Do not commit a second copy of the architecture under cache/ and then import it from there. If the runner needs Python modules, those modules live under worldfoundry/ and get reviewed with the rest of the PR.

Register the route. The simplest form is a pipeline_target on the catalog YAML. If several aliases or a runtime profile should share one class, add or update a binding under worldfoundry/data/models/bindings/pipelines/ and let bindings.py resolve it. The generic benchmark runner adapts pipeline calls in worldfoundry/evaluation/models/pipelines/invocation.py. Update that adapter only when the new pipeline needs an invocation form it does not already support. A working worldfoundry-eval run <model-id> path is the milestone.

Write or update the catalog entry last, after the class names exist, so the YAML cannot claim a target that does not import. Record source URLs, license, Hugging Face repo_ids, local checkpoint roles, tasks, runtime_profile, and an honest integration.status. Use planned until a check report exists. Use blocked when a missing license, checkpoint, API key, or GPU class prevents a run. Use integrated only after the required checks pass and the report hashes still match the current manifest.

If the official demo has a published command line you can compare against, add a small model-zoo check helper. Add a focused regression under tests/ (manual streaming demos live in tests/stream/) for the deterministic parts: request shaping, path resolution, and “missing checkpoint raises.” Do not gate the PR on a multi-hour video run unless that is the only way the route can be shown.

Catalog status is a claim, evidence is the proof

source.status describes how the weights and code are obtained. open_source means official public source exists and license metadata is recorded. api means a hosted provider or private service is the only execution surface. closed means there is no public runnable source or checkpoint. Those labels are about provenance, not about whether WorldFoundry can run the model today.

integration.status is the WorldFoundry-side claim. planned means source or code is in tree but the runnable checks are incomplete. blocked means a concrete condition — license, checkpoint, dependency, API, or hardware — stops the run. integrated means the required checks passed and the report is still fresh for the current manifest hash. If you bump the YAML without regenerating the report, the support matrix treats the evidence as stale and will not agree with the label.

Discover, stage, and run

After the YAML is in the catalog tree, confirm the zoo can see it. From the unified environment:

worldfoundry-eval zoo models --json
worldfoundry-eval zoo model-show --model-id <model-id-or-alias> --include-manifest --json
worldfoundry-eval zoo model-specs --model-id <model-id> --json

model-show is the fastest way to see whether aliases, pipeline_target, checkpoint roles, and the runtime profile parsed the way you intended. model-specs is the compact view of what a runner will actually consume.

Staging checkpoints is separate from claiming them. --check-local only reports whether the configured cache already has the files:

worldfoundry-eval zoo model-download --check-local --model-id <model-id> --json

On a machine that is allowed to fetch weights, add --execute and, if you need to pin the destination, --repo-id and --cache-dir. Write a download report with --report-path tmp/model_zoo/download/<model-id>.json so a reviewer can see what landed without copying blobs into git. Flags such as --disable-xet, --disable-hf-transfer, --timeout, and --retries belong only on that fetch machine.

A first end-to-end inference check uses the model runner, not a benchmark:

worldfoundry-eval evaluate \
  --mode model \
  --model-id <model-id> \
  --model-manifest-dir worldfoundry/data/models/catalog \
  --requests-path tmp/demo/requests.jsonl \
  --output-dir tmp/model_eval/<model-id> \
  --metric artifact_count \
  --json

artifact_count only asks “did a file appear.” That is enough to prove the route. Scoring with a real benchmark protocol is a later PR, and it must consume the GenerationResult artifacts rather than calling the pipeline from inside a metric. If you are still choosing an environment, start from Run inference and the unified bootstrap; if the model needs a dedicated conda profile, install it with bash scripts/setup/model_env_install.sh --model <model-id> instead of guessing a prefix.

What the check report must contain

The model check command writes under tmp/model_zoo/validation/<model-id>/. The report is the object the support matrix hashes. It should record the manifest entry hash, the check-script hash, the git commit, per-stage results, any blockers, and the next action. If the YAML moves and the report does not, readiness falls back even when integration.status still says integrated.

Treat that directory as evidence, not as a cache of weights. Put checkpoints under ${WORLDFOUNDRY_CKPT_DIR} or the Hugging Face cache; put generated videos under tmp/ or ${WORLDFOUNDRY_ARTIFACT_DIR}.

What a reviewer needs in the PR

A reviewer should be able to rerun your path without asking which GPU node you used or which hidden env var made it work. Attach the catalog YAML diff, tmp/model_zoo/validation/<model-id>/report.json, the cache location and Hugging Face repo_id (not the weights), the official demo command with raw logs and the artifact it produced, the WorldFoundry runner command with its output path, and the focused tests. When you promote status, refresh tmp/preflight_report.md so the support matrix and the PR tell the same story.

Do not commit checkpoints, generated videos, local caches, API keys, or private assets. Those belong in the report as paths and hashes.

Cache and runtime hygiene

Cache directories are for temporary downloads, checkpoint copies, dataset mirrors, official-runtime clone caches, generated previews, and logs. They are not a second worldfoundry/ tree. An in-tree runner must not import Python modules from a cache path. If the official repo is cloned only to harvest inference.py, vendor the infer-only slice under the appropriate package and delete the training side.

Transforms that actually run at inference time may stay in the runtime package — GroundingDINO util.transforms, DUST3R crop helpers, DROID-SLAM RGB-D geometry. Training datasets, dataloaders, losses, callbacks, augmentation-only modules, and experiment scripts should not. pixelsplat is the current exception under 3D base models because it still uses the official Lightning test runner and a dataset-shaped loader. Treat that as debt to port into a pure inference runner, not as a pattern for the model you are adding.

Model homepage

Each model article lives in its own file: docs/fumadocs/content/docs/guides/supported-models/<model-id>.mdx and the paired .zh.mdx. The Models catalog still links to /docs/guides/supported-models/<model-id>.

npm --prefix docs/fumadocs run models:generate writes an initial page from the catalog docs: block and the recipe JSON. Edit that file for model-specific narrative — overview, architecture, figures, extra sections. Shared widgets stay embeddable:

• <ModelCommandBuilder modelId="<model-id>" locale="en" /> — run-command box
• <ModelRelatedRecipes modelId="<model-id>" locale="en" /> — same-family links

To keep hand edits, set pageSource: authored in the frontmatter. Generated pages keep pageSource: generated and will be refreshed on the next generate. YAML docs.overview only seeds a generated page; it does not render the article once you own the MDX file.