Validation

Introduction

Validation evidence should be proportional to the changed surface: small enough to run locally, specific enough to catch the regression, and explicit about skipped heavy stages.

Evidence by change type

• Docs only — npm run types:check, npm run build, affected route returns 200. Extra when promoted: Browser screenshot for layout or interaction changes.
• CLI or runner — Public CLI discovery plus a representative evaluate command. Extra when promoted: Existing-results and model-mode runs if output contracts changed.
• Model manifest — zoo model-show, zoo model-download --check-local. Extra when promoted: Clone, checkpoint, demo parity, runner parity, benchmark run.
• Model runtime integration — In-tree runtime import, no machine-local source checkout dependency, runtime profile with backend_stage, and focused regression. Extra when promoted: Checkpoint-backed run and one normalized benchmark scorecard.
• Benchmark manifest — zoo benchmark-show, preflight runtime --profile <benchmark-id>. Extra when promoted: Clone, dataset, official runtime, normalized scorecard.
• Benchmark runner — zoo benchmark-run --mode official-validation or --mode official-run. Extra when promoted: Official result import, official runtime, and full-suite fairness gate.
• Suite or matrix — run --all-benchmarks --plan-only --json. Extra when promoted: Plan selected cells, then validate artifacts with runnable benchmark paths.

Public tests

All first-party test sources live under tests/ and are published with the repository and source distribution. Tests use the repository's Apache-2.0 license. The wheel contains the runtime packages.

Public CPU gate

From the repository root, with Python 3.10 or newer:

python -m pip install -e ".[test]"
make test
# Individual development tests can be selected with pytest:
python -m pytest -q tests/runtime/test_geometry_regression_impact.py

The default selection in pyproject.toml covers process logging, GPU lease bookkeeping with simulated devices, checkpoint staging, job persistence, Conda dispatch, pipeline cleanup, job shutdown, and packaging boundaries. It runs without CUDA, checkpoint downloads, or provider API credentials. CI uses the complete selection plus the coverage, evidence, historical-reference and JUnit-validator contracts. make test-infer is an alias; use PYTHON=python3 on systems whose python command still points to Python 2.

The Make targets collect an independent complete node manifest before running tests. They reject skips, collection errors, missing or additional nodes, duplicates, source changes and stale reports. TEST_ARGS applies only to the execution; filtering out nodes with -k or -m fails the gate. Public CPU and CPU Torch manifests and JUnit results are uploaded by CI. Successful standard library subtests keep their parent node; failed or skipped subtests still fail.

Extended suites

Select an extended suite or a single test explicitly:

python -m pytest -q tests/eval_core/test_evaluate_runner.py
python -m pytest -q tests/studio/test_conda_dispatch.py
# Collect the extended automated suites with their dependencies installed:
python -m pytest --collect-only tests

Extended tests can require additional extras (for example eval_core, ui, or a model's runtime environment), local benchmark helpers, assets, or CUDA. Some historical regression tests refer to private training/benchmark code or earlier repository layouts. Their sources remain available here; the public CPU gate is the supported CI baseline. Installing the test extra alone does not provision those resources.

manual/ and stream/ are excluded from recursive pytest collection. Run their scripts directly after preparing the relevant model environment:

PYTHONPATH=. python tests/manual/test_wan_2p2.py
PYTHONPATH=. python tests/stream/test_wan2p2_stream.py

These scripts can download weights, use GPU memory, call real provider APIs, and write output artifacts. Supply credentials through the provider's environment variables. Real-weight MiniMax H3 tests use MINIMAX_H3_CKPT_DIR, defaulting to ~/.cache/worldfoundry/checkpoints/minimax-h3-components.

Fast inference contracts and CUDA checks

The tensor suite needs only the public test dependencies, CPU Torch, safetensors and einops. It runs without checkpoints, renderers or a GPU:

python -m pip install "torch==2.7.1" --index-url https://download.pytorch.org/whl/cpu
python -m pip install -e ".[test]" "safetensors>=0.4" "einops>=0.7,<0.9"
make test-infer-tensors

In a prepared 3D model environment, also run the result serializers and CUDA contracts. The serializer imports use that environment's model and rendering dependencies, including plyfile and gsplat:

make test-infer-contracts PYTHON=/path/to/model-env/bin/python
make test-infer-cuda-contracts PYTHON=/path/to/model-env/bin/python

The first target uses small tensors and synthetic geometry without model weights. It covers real checkpoint round trips and meta-device assignment, rejection before partial mutation, remap collisions, camera conventions, masked PLY and array exports, fresh-process result serialization, request recovery, and concurrent submission. The CUDA target checks producer-event ordering, pinned-memory copies, fallbacks, and draining queued work after a callback fails. It fails when CUDA is unavailable rather than reporting a wholly skipped GPU suite as success.

After setting the regression checkpoint and input roots described below, a short real Pi3X test checks a reused model against a fresh model, with changed inputs, an intervening failed request, and exact repeatability:

python -m pytest -q tests/pipelines/test_pi3x_real_inference.py

This is a state-isolation check, not an accepted-output quality baseline. The fixed-weight numerical replay gate remains necessary for changed model behavior.

Video generation and world models

make test-infer-tensors includes the video component numerical contracts; make test-infer-video-tensors selects just this part. Real small Wan networks compare fused QKV and static cross-KV caches against the original network over different batches, temporal/spatial shapes, noncontiguous layouts and A/B/A requests. Causal VAE convolution compares chunked execution against independent full-sequence convolution and checks that future frames cannot affect earlier outputs. Sampling and CFG use analytic endpoints, with UniPC history reset, failure cleanup and global RNG isolation. Explicit zero or negative dimensions, frame counts, steps and fps fail before inference; only None selects defaults.

LTX's timestep-conditioned VAE also injects noise during decoding. Video, audio/video and request-based tensor decoders use a local generator on the latent device, seeded from the request. Small real VAE tests cover tiled and untiled decoding, A/B/A seeds, global RNG isolation and exact agreement with the original VAE given the same random state. Untiled tensor decoding calls the decoder directly. The former uncontrolled LTX RGB output is stochastic; accepting a seeded reference requires separately recording this behavior fix and proving latent and controlled-decoder parity, rather than silently replacing a failed reference.

The existing tests/manual/geometry_regression_cases.json now includes geometry, video and world-model cases. Legacy script names and GEOMETRY_* commands remain available; plan-inference, replay-inference and verify-inference expose the broader scope. New short cases declare Wan 2.1 1.3B, Wan 2.2 TI2V 5B, LTX 0.9.8 I2V, CogVideoX 2B, Cosmos Predict 2.5 2B, Matrix Game 2 and DreamX World 5B AR. LTX retains its native two-pass ten-step recipe. These cases compare available latents, raw frames, decoded RGB exports and fps; they check changed conditions, A/B/A requests, expected failures, causal continuation, reset and replay. They protect these concrete inputs and state transitions, not full-demo quality.

Additional cases cover Cosmos Transfer 2.5's general/edge 2B checkpoint, Gamma's released four-step causal checkpoint, and Oasis 500M's video prompt route. Gamma uses 189 frames per view, as required by the released weights; the case checks two 320×480 views and recovery after a rejected short request. Transfer samples its minimum 93-frame window (24 latent frames) before cropping the requested 17 RGB frames. Stage control.mp4 under WORLDFOUNDRY_REGRESSION_INPUT_ROOT with at least 17 frames (a longer clip also exercises bounded conditioning reads). Place the official buildHouse_flat sample's first_frame.png, prompt.txt, action_left.json and action_right.json under gamma-world/buildHouse_flat. Oasis expects open-oasis/context.mp4 with multiple frames and open-oasis/forward.one_hot_actions.json with two 25-column rows: set column 11 (forward) to 1 and every other column to 0. Its adapter reads one prompt frame and holds the last action through six output frames. Checkpoint paths and encoder/tokenizer assets are explicit in each case. These declarations select required replays; they do not certify GPU execution or other variants.

Video cases can replace call with sequence. Each step specifies a unique name, method (default __call__), call and selected outputs. Use outputs: [] for session configuration or reset. expect_error requires an exact qualified exception type and message pattern; unrelated errors fail. Returned CPU buffers are copied immediately, preserving earlier responses. comparisons specifies exact equal or different relationships within one run. array_contracts checks shape, dtype, ranges or exact values, including the requested fps numerator/denominator. export_artifacts selects MP4s inside the run directory for full decode and comparison. Reference/candidate audits still require exact numerical equality.

For a generator that returns only a video path, output_arrays can select an MP4, WebM, MOV or MKV inside the run directory. The replay decodes every frame and saves artifact.<relative-path>.frames and .fps in arrays.npz, so required outputs cannot be satisfied by a path or plan alone. Also declare the video in export_artifacts to compare the encoded artifact's decoded pixels and frame rate against the reference. Oasis uses this route because its subprocess returns an MP4 without in-memory tensors; its generator fixes the random seed to 0.

make coverage-inference PYTHON=python3
make plan-inference PYTHON=python3 INFERENCE_BASE=<comparison-commit>
make replay-inference PYTHON=/path/to/prepared-env/bin/python \
  INFERENCE_PROFILE=/private/path/inference-regression.json
make verify-inference PYTHON=python3 \
  INFERENCE_PROFILE=/private/path/inference-regression.json \
  INFERENCE_REPORT=/private/path/completed-run/report.json

Coverage inventories catalogued native video/world bindings and registered variants of their pipeline classes. A shared class, family, alias or historical verified label does not assign a case to another model ID. case_defined means a recipe exists; it does not imply accepted references or successful GPU inference. A frozen baseline preserves existing gaps while CI rejects new uncovered identities, loss of an existing recipe and affected native variants without a runnable numerical replay. Releases requiring complete coverage can invoke the coverage script with --require-complete; gaps return exit 2.

Runtime profile ownership uses the declared model ID and pipeline binding, including source_repos[].in_tree_path for code executed in subprocesses. These declarations supplement static imports. Unknown paths still expand selection conservatively and fail when an affected native variant lacks a real checkpoint replay.

Appending recipes and their component declarations selects the added cases only when the exact committed before/after definitions prove that every existing recipe, ownership and global rule is unchanged. Changed test orchestration additionally requires unchanged committed runtime, environment and asset files, matching checkout contents, its CPU contracts and a real MG2 smoke replay. Changed production code remains independently selected; dirty definitions, deletions or reclassifications keep the full replay selection.

The legacy InSpatio, LingBot Cam/Act/V2 and ATI cases each use their own released checkpoint. Set WORLDFOUNDRY_REGRESSION_INSPATIO_SCENE_ROOT to the accepted scene containing video_example_00/input/{video.mp4,target_tcw.txt}. WORLDFOUNDRY_REGRESSION_LINGBOT_CONTROLS contains 21-frame poses.npy and intrinsics.npy, plus action.npy for Act. ATI requires its fixed quantized packed-NPZ track file in WORLDFOUNDRY_REGRESSION_ATI_TRACKS. Inputs are hashed alongside weights. These short numerical replays do not certify full-demo quality or unrelated variants.

verify-inference checks that a trusted GPU-host report matches this commit, source tree, matrix, dependency declarations and pinned reference index, that all affected cases executed, and that every numerical field is exact. It requires a --base commit-diff plan and recomputes the changed paths; manually listed or omitted paths cannot bypass the affected-case gate. Preflight, stale commits, omitted cases, relaxed tolerances and failed runs cannot pass. Public CPU and tensor gates must pass separately.

The hosted inference-replay job consumes a successful official Actions artifact from a maintainer-dispatched control workflow. A missing, expired, foreign or stale receipt fails. The dispatch verifies the candidate's exact commit, tree, comparison base and pinned reference index before publishing trusted-inference-evidence. Evidence-only dispatches run the attestation job; ordinary CI runs verify the receipt. This workflow uses a prepared trusted GPU host without requiring a hosted GPU runner. Configure the repository variables INFERENCE_EVIDENCE_CONTROL_REF, INFERENCE_EVIDENCE_WORKFLOW_PATH and INFERENCE_REFERENCE_INDEX_SHA256; protect the merge branch with the inference-replay and CPU checks. A workflow declaration alone does not enable branch protection. Uncovered and unexecuted variants remain gaps.

MG2 changes, including changes to its numerical contract, additionally require all six real-image, 12-block trajectories against a frozen accepted revision. The reference's complete digest is pinned. Flow/x0 outputs, every rolling cache, conditions, latent/video/RGB pixels, VAE state and continuous RNG histories must match exactly. Two current paths agreeing with each other is insufficient. This independent contract accompanies the public-pipeline A/B/A replay; neither can stand in for the other. The gate never regenerates accepted references.

After committing the candidate, run both GPU gates on the prepared host. Build the bounded dispatch input from their completed reports:

INFERENCE_REFERENCE_INDEX_SHA256=<pinned-index> python3 tests/manual/inference_actions_evidence.py compact \
  --base <comparison-commit> --head <candidate-commit> \
  --conditioned-report /private/path/mg2-conditioned.json \
  --junit /private/path/mg2-conditioned.xml \
  --generic-replay /private/path/replay/report.json --output /private/path/evidence.json

Dispatch CI on the configured control branch with those two commit IDs and the compact JSON as source_revision, base_revision and evidence_json. After that dispatch succeeds, retry the candidate's inference-replay job. The companion inference-evidence-verified.yml performs this retry once it is installed on the repository's default branch; until then retry it explicitly.

Validation while optimizing

Use the latest successful replay's source_hashes to help identify affected cases, together with explicit component/capability declarations and static dependencies. Runtime traces only cover paths that ran: they do not cover an unexecuted branch or a dynamically selected backend. Treat an unresolved impact as uncovered and expand the affected component's tests. Keep the reference implementation available for differential tests and safe fallback. Separate exact optimizations from approximations; specify numeric and task-quality limits for the latter and never silently widen tolerances or regenerate accepted references.

GitHub CI runs the public CPU, CPU Torch tensor and packaging gates on pull requests, pushes to main and release/**, and manual dispatch. Result serializers, CUDA contracts, real-weight replays and broader model benchmarks require the prepared model environment. Hosted CI requires their official evidence for affected inference changes and publishes the affected-case plan. Daily GPU replays are available through the explicit host schedule described below.

Automatic affected-case selection

make plan-geometry PYTHON=python3 GEOMETRY_BASE=<comparison-commit>
# Optional: add actual imported-source evidence from a successful matrix.
python3 tests/manual/geometry_regression_impact.py --base <comparison-commit> \
  --evidence /absolute/path/to/successful-runs --output tmp/3d-regression-plan.json

The planner combines the explicit component declarations in tests/manual/geometry_regression_dependencies.json, conditional and relative Python imports, literal lazy imports, package initializers, and optional recorded imports. Git rename and deletion handling includes the old paths. Shared inference infrastructure or matrix changes select all cases; new cases require dependency declarations. Unknown paths or an unresolved import graph select the full matrix and exit with uncovered, requiring the coverage gap to be resolved. A docs-only change can report no_inference_changes; it still requires the CPU gates.

Only changes to the planner and source-snapshot orchestrator have a narrower verification rule: immutable Git entries must prove that all runtime code, environments, assets, matrix and dependency definitions are identical. The complete commit diff and inspected tooling must match the committed blobs. Those changes require the full CPU contracts and both MG2 GPU gates to exercise the verification chain. Missing history, changed case definitions or a mixed production change retains the full conservative model selection.

CI uploads geometry-impact even when planning fails. This artifact lists required replays; it is not a successful GPU result. The runner rejects stale tree/policy hashes, contradictory empty plans, or plans omitting cases required by the current declarations and import graph.

Scheduled GPU regression on a prepared host

Use the same model Python, accepted assets and environment as the numerical reference. First copy explicitly accepted outputs into durable private storage:

python3 tests/manual/geometry_regression_schedule.py import-reference \
  --source /absolute/path/to/accepted-runs \
  --destination /absolute/path/to/private-state/reference

The importer preserves the original evidence, relocates exported artifact paths, verifies exact numerical equality, and records each copied file's SHA-256. It refuses an existing destination. Save the returned reference_index_sha256 in a private host profile such as ~/.config/worldfoundry/geometry-regression.json:

{
  "source_root": "/absolute/path/to/WorldFoundry",
  "source_ref": "refs/heads/your-validated-branch",
  "python": "/absolute/path/to/model-env/bin/python",
  "matrix": "tests/manual/geometry_regression_cases.json",
  "dependencies": "tests/manual/geometry_regression_dependencies.json",
  "reference": "/absolute/path/to/private-state/reference",
  "reference_index_sha256": "<SHA-256 returned by import-reference>",
  "state_root": "/absolute/path/to/private-state",
  "cuda_visible_devices": "0",
  "env": {
    "WORLDFOUNDRY_REGRESSION_CHECKPOINT_ROOT": "/accepted/checkpoints",
    "WORLDFOUNDRY_REGRESSION_INPUT_ROOT": "/accepted/multiview-frames",
    "WORLDFOUNDRY_REGRESSION_INPUT_IMAGE": "/accepted/input.png",
    "WORLDFOUNDRY_REGRESSION_PANORAMA": "/accepted/panorama.jpg",
    "WORLDFOUNDRY_REGRESSION_LYRA_INPUT_IMAGE": "/accepted/lyra-input.jpg"
  },
  "case_timeout_seconds": 1800,
  "suite_timeout_seconds": 7200,
  "retain_runs": 7
}

The optional gpu_holder object configures an existing gg script using an absolute script, string-list start_args, its env (including GG_PYTHON), and optional log ready_patterns. The runner stops that script for replays and restarts it in a detached session after success, failure, timeout or interruption. If replay descendants cannot be stopped or GPU leases cannot be released, it fails and defers holder restoration until those resources are cleared.

Preserve required runtime variables such as LD_LIBRARY_PATH and CUDA_HOME in the private profile's env. Cron jobs retain the prepared tool PATH. A CUDA preflight compares Python, Torch, CUDA, cuDNN, GPU and package versions with all selected references before loading weights or stopping the holder; changed environments fail rather than weakening the comparison.

Models accepted in different Python/Torch environments use private case_environments overrides, for example "case_environments": {"flashworld": {"python": "/absolute/path/to/flashworld-env/bin/python"}, "worldfm": {"python": "/absolute/path/to/worldfm-env/bin/python"}}. Each override can also provide env, merged over the default profile environment. Every distinct selected environment gets its own preflight and each case runs in its configured interpreter. Unknown case IDs are rejected.

# Validate the runner with one case before enabling a complete daily matrix.
python3 tests/manual/geometry_regression_suite.py \
  --profile ~/.config/worldfoundry/geometry-regression.json --preflight-only
python3 tests/manual/geometry_regression_suite.py \
  --profile ~/.config/worldfoundry/geometry-regression.json --case-id pi3x
python3 tests/manual/geometry_regression_schedule.py install \
  --profile ~/.config/worldfoundry/geometry-regression.json --hour 3 --minute 0
python3 tests/manual/geometry_regression_schedule.py status \
  --profile ~/.config/worldfoundry/geometry-regression.json
# Remove only WorldFoundry's marked cron block.
python3 tests/manual/geometry_regression_schedule.py remove \
  --profile ~/.config/worldfoundry/geometry-regression.json

Installation preserves other cron entries and uses Asia/Shanghai. It copies controllers from the committed source revision; reinstall after changing those controllers. Each run snapshots the configured local committed ref, without pulling, resetting, or executing dirty source files. Separate model processes share the host's physical-GPU lease locks, have per-case and suite time budgets, and undergo exact numerical and imported-source audits. The complete short matrix runs daily; an on-demand --plan can select affected cases instead. latest.json records running before work starts and the final outcome; run logs, source revision and comparisons live under runs/. Retention keeps seven run directories by default and archives older compact reports under history/. Accepted references are outside disposable runs and are never regenerated by the runner. Profiles, local assets, weights and evidence stay outside Git. --preflight-only checks all selected assets and environments without loading model weights; its preflight_passed status is not numerical inference evidence.

Pin numerical environment settings in the host profile too, including OMP_NUM_THREADS and MKL_NUM_THREADS; CPU depth reconstruction can change with different thread counts even when every package version matches. Give each Python/Torch/CUDA/C++ ABI combination its own TORCH_EXTENSIONS_DIR, and ensure the extension compiler uses Torch's C++ ABI. A fresh cache alone does not fix an incompatible compiler default. DreamX honors explicitly enabled deterministic algorithms by disabling cuDNN benchmarking, which can otherwise select different convolution algorithms in separate processes. Keep references recorded under a changed determinism policy separately, with an original-production parity check; never overwrite existing references or relax tolerances to hide drift.

These short cases cover declared inputs and capabilities. The Lyra-2 case still disables reconstruction. Full demos, quality benchmarks, and tests of new backends remain necessary before changing integration or release status.

Docker helper

WORLDFOUNDRY_DOCKER_GPUS=none tests/run_tests_docker.sh
tests/run_tests_docker.sh tests/eval_core/test_cli_ux.py
WORLDFOUNDRY_DOCKER_EXTRAS=test,eval_core tests/run_tests_docker.sh make:test-eval-core

The helper mounts the checkout and local caches, installs the selected extras, and runs the requested target. Its default target is the public CPU gate.

Adding tests

Place deterministic regressions beside the relevant module tests and store small text fixtures under that suite. Move scripts that run inference at import time into manual/ or stream/. Use environment variables for local checkpoints and skip real-weight tests when their resources are unavailable. Bytecode, caches, logs, generated output directories, secrets, and weights remain excluded from version control and source releases.

Preparing the optional CPU eval-core suite

Install Torch and Torchvision from the official CPU wheel index first, then install the declared test extra from the normal package index:

python -m pip install --index-url https://download.pytorch.org/whl/cpu \
  "torch>=2.7,<2.12.0" "torchvision>=0.22,<0.27.0"
python -m pip install -e ".[eval_core]"

The separate first command is intentional: Python package metadata cannot select a CPU-specific wheel index, while pointing the whole extra at the PyTorch index would hide packages such as pytest and NumPy.

Numerical regression for geometry, video and world models

The CPU gate and mocks do not prove that inference with real weights remains correct. After changing loading, preprocessing, inference, camera conversion, or export code, replay an accepted case in separate processes using tests/manual/geometry_regression.py. The matrix in tests/manual/geometry_regression_cases.json covers Depth Anything, Pi3/Pi3X, VGGT, CUT3R, LoGeR, LingBotMap, Hunyuan Mirror, HY-WorldMirror 2.0, FlashWorld, and WorldFM. Its additional Lyra-2 case validates video generation and camera trajectories only, with 3D reconstruction disabled. Prepare the model environment and local assets separately.

git worktree add --detach /tmp/worldfoundry-3d-reference <validated-revision>
export WORLDFOUNDRY_REGRESSION_CHECKPOINT_ROOT=/absolute/path/to/checkpoints
export WORLDFOUNDRY_REGRESSION_INPUT_IMAGE=/absolute/path/to/accepted-image.png
# The multiview cases use frame-0.png, frame-16.png, and frame-32.png.
export WORLDFOUNDRY_REGRESSION_INPUT_ROOT=/absolute/path/to/accepted-sequence
# Set REGRESSION_GPU_ID to an available GPU index or UUID from nvidia-smi.
export CUDA_VISIBLE_DEVICES="${REGRESSION_GPU_ID}"

# Run all three commands in the same model-specific Python environment.
python tests/manual/geometry_regression.py run \
  --case tests/manual/geometry_regression_cases.json --case-id pi3x \
  --source-root /tmp/worldfoundry-3d-reference --output-dir tmp/3d-replay/reference/pi3x
python tests/manual/geometry_regression.py run \
  --case tests/manual/geometry_regression_cases.json --case-id pi3x \
  --source-root . --output-dir tmp/3d-replay/candidate/pi3x
python tests/manual/geometry_regression.py compare \
  --reference tmp/3d-replay/reference/pi3x --candidate tmp/3d-replay/candidate/pi3x \
  --report tmp/3d-replay/comparison.json

After replaying every affected case, audit coverage and check that the evidence still matches the current source:

make test-geometry PYTHON=/path/to/model-env/bin/python \
  GEOMETRY_REFERENCE=tmp/3d-replay/reference \
  GEOMETRY_CANDIDATE=tmp/3d-replay/candidate

Each root must contain a run directory named for every matrix case ID. Missing cases, changed case parameters, and changed imported-source hashes fail the gate. GEOMETRY_MATRIX can select the affected case matrix; use the complete matrix for changes to shared inference components. This audits completed real inference evidence and does not replace the replay commands.

Each run records input and checkpoint SHA-256 hashes, case parameters and seed, dependency versions, CUDA/precision settings, hashes of imported source files, and raw numerical outputs. Comparison requires identical inputs, weights, parameters, and environments. It checks output fields, shapes, dtypes, finite values, and elementwise values for depth, points, confidence, masks, and cameras. Cases with save also compare exported PLY values, images, and NumPy artifacts. Missing assets, failed inference, empty outputs, NaNs, missing fields, and modified evidence fail. Reference directories are never overwritten automatically.

Comparison is exact by default. Cases can set deterministic: true to require deterministic Torch algorithms and configure cuBLAS before CUDA initialization; unsupported deterministic operations fail. The Hunyuan Mirror and WorldFM cases use this setting. WorldFM also pins the inference seed after model loading; a seed set only before loading can be affected by initialization. Its panorama uses WORLDFOUNDRY_REGRESSION_PANORAMA, and Lyra-2 uses WORLDFOUNDRY_REGRESSION_LYRA_INPUT_IMAGE.

For nondeterministic GPU operations, repeat the reference run first, measure its variation, and justify explicit floating-point --atol / --rtol settings. Boolean masks and integers remain exact. The matrix audit remains exact and does not inherit tolerances from a separate comparison report. Do not widen tolerances or replace an accepted reference with failing output to hide regressions.

Additional cases declare target (module:Class), load, call, assets, and required_outputs. ${OUTPUT_DIR} resolves to the run directory; use environment variables for other paths. extra_calls can capture raw depth, and capture_modules can capture raw forward outputs when a pipeline only returns artifact paths. Generative cases must also pin prompts, trajectories, sampling steps, and precision, and review rendered output quality.

Validation applies to specific code, weights, inputs, and environments. Changes to shared components require replays of every affected model and variant; unexecuted cases remain uncovered. GitHub CI runs the public CPU gate; real inference must run separately on a GPU host with local weights and pass make test-geometry before accepting an inference change. Numerical parity protects accepted cases against regression, but cannot prove correctness for arbitrary inputs or replace benchmarks with ground-truth depth and calibrated cameras. Accept a reference using official results, quality metrics, and human review before using it as a regression baseline.

Release gate

From a lightweight editable checkout, install the declared gate dependencies before running the core commands:

python -m pip install -e ".[dev,test]"

make test
make lint
make compile-eval
make cli-entrypoint-check
make cli-check
conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval evaluate --results-path tmp/results.jsonl --output-dir tmp/worldfoundry_evaluate --json
conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval zoo benchmarks --json

cd docs/fumadocs
npm run models:check
npm run types:check
npm run build
curl -I http://localhost:8013/docs
curl -I http://localhost:8013/zh/docs

conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval zoo model-show --model-id <model-id> --include-manifest --json
conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval zoo model-download --model-id <model-id> --check-local --json

conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval zoo benchmark-show --benchmark-id <benchmark-id> --include-spec --json
conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval zoo benchmarks --json
conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval preflight runtime --profile <benchmark-id> --manifest worldfoundry/data/benchmarks/runtime_profiles/official/<benchmark-id>.yaml --output-dir tmp/preflight/runtime/<benchmark-id> --json
conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval zoo benchmark-run --benchmark-id <benchmark-id> --mode official-validation --official-results-path <official_results.json> --generated-artifact-dir <generated_artifacts> --output-dir tmp/benchmark_zoo/official_validation/<benchmark-id> --json

Artifact acceptance

• run_manifest.json — Any evaluation run. Accept when: Command, ids, status, timestamps, counts, and output paths match the run.
• execution_plan.json — Runner or suite changes. Accept when: Resolved model, benchmark, metrics, and requests are deterministic.
• sample_ledger.jsonl — Partial-failure review. Accept when: Failed/skipped samples include visible reasons.
• metrics/summary.json — Metric changes. Accept when: Aggregates, failed counts, and metric keys match the expected schema.
• scorecard.json — Publishable evaluation evidence. Accept when: Eligibility, leaderboard state, metrics, and blockers are present.
• report.json — Model/benchmark validation. Accept when: Stage status, command, repo/checkpoint/dataset refs, and blockers are recorded.
• suite_report.json — Suite or matrix changes. Accept when: Every selected cell is pass, fail, or skipped with a reason.

Integration claim rule

integrated is reserved for a reviewable implemented support surface, not for a source URL or plan alone. For model runtimes, the architecture or runtime glue, launch path, environment contract, checkpoint/input/output contract, and minimum real inference evidence must be reviewable from the repository plus declared assets. For benchmarks, an in-tree runner, official command, isolated importer, or official-result normalizer may be integrated when its CLI path, artifact/data requirements, and scorecard output are evidenced.

Do not use integrated as a leaderboard synonym. Normalizer-only benchmark integrations must keep verification_status=normalizer_only and leaderboard_valid=false until the full official scorer or judge has run and the exact scorecard proves eligibility.

Runner evidence

Failure policy

• Pass — Required stages ran and produced expected artifacts. Release treatment: Can support the matching claim.
• Partial — Some stages were skipped because they require network, credentials, data, GPU, or license review. Release treatment: Usable for development, not for stronger public claims.
• Blocked — A required source, dataset, license, dependency, checkpoint, or API condition is unavailable. Release treatment: Keep the manifest blocked or planned.
• Stale — Report hash, manifest hash, git commit, validator script, or timestamp no longer matches. Release treatment: Rerun validation before counting it.
• Runner-only — Schema and runner path pass, but official runtime did not run. Release treatment: Not leaderboard evidence.

Validation matrix

The matrix is the release gate for model-zoo and benchmark-zoo readiness. It merges manifest entries with staged validation reports and recomputes integration status.

conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval run \
  --all-benchmarks \
    --plan-only \
  --output-dir tmp/worldfoundry_plan_matrix \
  --json

conda run -p "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}" worldfoundry-eval validate-artifact \
  tmp/worldfoundry_plan_matrix/suite_manifest.json \
  --kind suite-manifest \
  --strict \
  --json

• accepted_integrated — Fresh report passes every required stage and manifest says integrated. Maintainer action: Keep evidence current.
• accepted_variant_integrated — One or more scoped variants pass while the parent family remains planned. Maintainer action: Continue validating remaining variants.
• validated_but_manifest_not_promoted — Report passes, manifest still says planned or blocked. Maintainer action: Promote only after artifact review.
• manifest_integrated_without_report — Manifest says integrated, but no fresh validation report exists. Maintainer action: Run staged validation or downgrade.
• manifest_overclaims_integration — Manifest says integrated but computed stages do not pass. Maintainer action: Downgrade or rerun full validation.
• report_overclaims_integration — Report claims integrated but staged evidence is incomplete. Maintainer action: Discard or rerun report.
• stale_report — Report hash, script, git commit, schema, or manifest entry changed. Maintainer action: Rerun validation.
• partial_or_blocked — Report exists but stages failed or were skipped. Maintainer action: Follow next_action.
• not_validated — Manifest entry has no report. Maintainer action: Run the matching runtime preflight and official validation run.

Checklist