---
title: Inference optimization
description: Which WorldFoundry inference speedups you can turn on, which combinations the runtime rejects, and how to confirm a knob actually ran.
---

<Callout type="warn" title="Experimental">
These optimizations are experimental. They may change, may not apply to every model, and should not be treated as a supported production contract.
</Callout>

WorldFoundry only treats a technique as available when a public control exists and the main path consumes it. An accepted flag is not a speedup. After a forward, read `requested / effective / fallbacks` from `runtime_optimization_report()`.

[Pluggable diffusion acceleration](/docs/guides/diffusion-acceleration) covers the Sol-Engine / FlashDreams comparison, native SANA/Wan adapters, removal and request isolation, and configurations accepted or rejected by complete video diagnostics.

Most named knobs below are on the **native Wan 2.1 / Wan 2.2** pipelines (`Wan2p1*`, `Wan2p2*`, including VACE). Wan 2.5 / 2.6 / 2.7 are hosted APIs and have no local kernel controls. Other native recipes share only a subset (dtype, offload, sometimes attention or VAE tiling). Family runtimes (LTX, CogVideoX, and the rest) keep their own flags — do not copy Wan keys onto them.

<Callout type="info" title="Accepted is not effective">
Missing kernels, calibration, process groups, or checkpoint weights fail at build time or report an explicit dense / SDPA fallback. Wrapper installation alone (`*-wrapper-installed (runtime-pending)`) is not acceleration.
</Callout>

## Default recommendation

<Callout type="idea" title="First run vs throughput">
**First run / official-looking output.** Keep the catalog defaults: `torch_dtype="bfloat16"`, `offload_mode="resident"` when VRAM fits (Studio default for Wan2.2 TI2V-5B), otherwise `async-block`. Leave feature cache, approximate attention, quantization, CUDA Graph, and TAEW preview off. Do not cut steps or CFG for an official benchmark.

**Throughput on the same checkpoint.** Stay resident BF16. Set `attention_backend` to a real Flash / Sage provider for your GPU, then `fuse_qkv=True`, `fused_rope=True`, and `static_cross_kv=True`. Add `compile=True` only after a warmup request — the first compiled forward is slow. Turn on Wan VAE spatial tiling / temporal chunks if decode OOMs or dominates latency. Add **one** feature-cache algorithm only if you accept skipped DiT steps.
</Callout>

## What you can do

These controls are wired. Enable them from `from_pretrained(...)`, Workspace **load JSON**, or `worldfoundry-eval run --pipeline.load.<kebab-name>` when that key is on the model’s load schema.

### Precision and placement

<KvCatalog heading="Placement" vars rows={[
  { name: "torch_dtype / dit_weight_dtype", hintLabel: "How", hint: "bfloat16 / bf16", description: "Run the DiT in BF16. FP32 checkpoints convert on load. This is the normal production precision, not an extra experiment.", reads: [{ label: "Applies to", value: "native Wan 2.1 / 2.2; other native recipes that pass dtype into RuntimePolicy" }] },
  { name: "offload_mode=resident", hintLabel: "Aliases", hint: "none / fast / resident", description: "Keep weights on GPU. Fastest when the model fits. Studio keeps Wan2.2 TI2V-5B resident on an H100-class card.", reads: [{ label: "Applies to", value: "native visual diffusion loaders that call parse_offload_policy" }] },
  { name: "offload_mode=async-block", hintLabel: "Aliases", hint: "block / cpu / layer", description: "Layerwise CPU masters with pinned H2D prefetch (at most two resident layers). Use only when VRAM is tight. The Wan VAE stays resident; only the DiT offloads.", reads: [{ label: "Caveat", value: "Incompatible with quantization and with device_map=balanced" }] },
  { name: "offload_mode=component", hintLabel: "How", hint: "offload_mode=\"component\"", description: "Whole-module CPU offload. Requires a vram_module_map on that component.", reads: [{ label: "Applies to", value: "components that declare a wrapper map" }] },
  { name: "device_map=balanced", hintLabel: "How", hint: "device_map=\"balanced\"", description: "Shard transformer layers across visible GPUs. Requires offload_mode=none so weights stay in VRAM.", reads: [{ label: "Applies to", value: "modules that declare layer_container" }] },
  { name: "inplace_residual", hintLabel: "How", hint: "inplace_residual=True", description: "In-place residual adds on the Wan DiT to cut activation memory. Inference-only; disabled under autograd.", reads: [{ label: "Applies to", value: "native Wan DiT only" }] }
]} />

### Attention backends

<KvCatalog heading="Backend" vars rows={[
  { name: "attention_backend", hintLabel: "Values", hint: "auto / flash / flash2 / flash3 / flash4 / sage / sage3 / xformers / sdpa / torch", description: "Dispatcher picks a kernel and reports SDPA if the provider is missing or the shape is ineligible. auto is the safe default.", reads: [{ label: "Applies to", value: "native Wan; HunyuanVideo also accepts attention_backend. Studio dropdown: auto, torch, flash_attn_2, flash_attn_3, sage, xformers" }] },
  { name: "WORLDFOUNDRY_ATTENTION_BACKEND", hintLabel: "Env", hint: "auto / flash / cudnn / efficient / math", description: "Process-wide SDPA policy for runtimes that call install_worldfoundry_inference_infra. Names such as flash_attention_2 map to auto unless you pin a real dispatcher backend on the pipeline.", reads: [{ label: "Related", value: "WORLDFOUNDRY_ENABLE_TF32 (default on), WORLDFOUNDRY_MATMUL_PRECISION=high, WORLDFOUNDRY_DETERMINISTIC=1" }] }
]} />

### Exact or numerically equivalent Wan kernels

<KvCatalog heading="Kernel" vars rows={[
  { name: "fuse_qkv", hintLabel: "How", hint: "fuse_qkv=True", description: "Merge Q/K/V projections. Optional qkv_strategy=auto|packed|split and qkv_split_threshold (Studio default 8192).", reads: [{ label: "Applies to", value: "native Wan DiT. Confirm fused_blocks and a real projection call after one forward" }] },
  { name: "fused_rope", hintLabel: "How", hint: "fused_rope=True", description: "Fused 3D RoPE. Safe to combine with Ulysses SP (the kernel takes the rank offset and padding). rope_precision is fp32 or fp64.", reads: [{ label: "Applies to", value: "native Wan DiT" }] },
  { name: "static_cross_kv", hintLabel: "How", hint: "static_cross_kv=True", description: "Reuse text cross-attention K/V for the request. Studio default for Wan2.2 TI2V-5B. Not compatible with CUDA Graph.", reads: [{ label: "Applies to", value: "native Wan DiT" }] }
]} />

### Compile and CUDA Graph

<KvCatalog heading="Compile" vars rows={[
  { name: "compile", hintLabel: "How", hint: "compile=True or torch_compile=True", description: "torch.compile on the bound forward. Optional compile_mode (default / reduce-overhead / max-autotune / max-autotune-no-cudagraphs), compile_backend, compile_fullgraph, compile_dynamic. First request pays compile + autotune cost.", reads: [{ label: "Also", value: "WORLDFOUNDRY_TORCH_COMPILE=1 for processes that use compile_module_if_enabled" }] },
  { name: "cuda_graph", hintLabel: "How", hint: "cuda_graph=True", description: "Capture and replay a stable-shape Wan DiT inner call. Off by default. With Inductor, compile_mode must be max-autotune-no-cudagraphs so graphs are not nested.", reads: [{ label: "Rejected with", value: "sequence parallel, feature cache, approximate attention, static_cross_kv, custom Inductor compile_options" }] }
]} />

### Parallelism

<KvCatalog heading="Parallel" vars rows={[
  { name: "sequence_parallel / sp_degree", hintLabel: "How", hint: "torchrun + sp_degree>=2", description: "Native Ulysses SP: exact attention, shards tokens across GPUs. Requires NCCL before load. Wan2.2 has 24 heads, so degree must divide 24 (2, 3, 4, 6, 8, 12).", reads: [{ label: "Rejected with", value: "feature cache, approximate attention, CUDA Graph" }] },
  { name: "cfg_parallel", hintLabel: "How", hint: "cfg_parallel=2", description: "Positive and negative CFG branches on two ranks. World size must match the SP×CFG mesh. Degree is only 1 or 2.", reads: [{ label: "Rejected with", value: "cfg_gate_step < 1.0" }] },
  { name: "vae_parallel", hintLabel: "How", hint: "vae_parallel=True or a degree", description: "Rank-parallel Wan VAE tiles with a typed all-gather. Degree must be the full SP×CFG world size when those are on. Needs torchrun WORLD_SIZE>1.", reads: [{ label: "Applies to", value: "official Wan VAE, not TAEW preview" }] }
]} />

### Wan VAE decode

<KvCatalog heading="VAE" vars rows={[
  { name: "vae_decode_autocast", hintLabel: "How", hint: "vae_decode_autocast=\"bf16\"", description: "Mixed-precision Wan decode. Slight numerical change vs FP32 decode.", reads: [{ label: "Applies to", value: "native Wan VAE" }] },
  { name: "vae_spatial_tiling", hintLabel: "How", hint: "vae_spatial_tiling=True", description: "Tile spatial decode. Aliases: vae_tiled_decode, or component tiled=True. Default tile 34×34, stride 18×16. A shape that yields one tile is not tiled execution.", reads: [{ label: "Not the same flag", value: "vae_tiling is Hunyuan / Cosmos Predict2. Wan ignores that name." }] },
  { name: "vae_temporal_chunk_size", hintLabel: "How", hint: "vae_temporal_chunk_size=4", description: "Causal temporal streaming decode. Convolution cache is kept across chunks. A sequence that fits in one chunk is not streaming.", reads: [{ label: "Applies to", value: "official Wan VAE" }] },
  { name: "vae_channels_last_3d", hintLabel: "How", hint: "vae_channels_last_3d=True", description: "Convert Conv3d weights to channels-last. Fails if the VAE has no Conv3d.", reads: [{ label: "Applies to", value: "native Wan VAE" }] }
]} />

HunyuanVideo and Cosmos Predict2 expose their own `vae_tiling` (and Cosmos tile size/stride) on those pipelines only.

### Distilled checkpoints

Wan2.2 public pipelines accept `peft_adapter_path` and merge a PEFT LoRA after an adapter/base/target audit. Quality and step count belong to that distilled checkpoint. There is no separate “4-step mode” without matching weights.

### Feature cache (lossy, one algorithm)

Opt-in, single-rank, mutually exclusive. Each CFG branch gets its own cache. TeaCache has published coefficients for Wan2.1 1.3B / 14B T2V and I2V variants, Wan2.1 Fun / VACE aliases, Wan2.2 Fun / A14B, and Wan2.2 TI2V-5B. Other model IDs must pass `coefficients` explicitly.

<KvCatalog heading="Cache" vars rows={[
  { name: "teacache", hintLabel: "How", hint: "feature_cache=\"teacache\" or teacache=True", description: "Skip DiT steps from a residual threshold (default 0.26) plus the model preset. Also teacache=<threshold>.", reads: [{ label: "Applies to", value: "native Wan denoisers with a preset or explicit coefficients" }] },
  { name: "adacache", hintLabel: "How", hint: "feature_cache=\"adacache\"", description: "Adaptive residual cache. Same SP / CUDA Graph bans as other caches.", reads: [{ label: "Applies to", value: "native Wan" }] },
  { name: "magcache", hintLabel: "How", hint: "magcache={\"ratios\": [...]}", description: "Requires calibrated per-step ratios. No Wan2.2 preset is published in-tree.", reads: [{ label: "Fails closed", value: "magcache=True without ratios" }] },
  { name: "blocktaylorseer", hintLabel: "How", hint: "feature_cache={\"algorithm\": \"blocktaylorseer\"}", description: "Pinned LightX2V per-block self / cross / FFN predictor. Default dense pattern [true, false, false, false].", reads: [{ label: "Applies to", value: "native Wan, single rank" }] },
  { name: "taylorseer", hintLabel: "How", hint: "feature_cache=\"taylorseer\"", description: "Legacy whole-stack residual predictor. Not LightX2V’s per-block algorithm. Default interval 4.", reads: [{ label: "Applies to", value: "native Wan, single rank" }] },
  { name: "custom", hintLabel: "How", hint: "feature_cache={\"algorithm\": \"custom\", ...}", description: "LightX2V Tea decision plus first-order stack residual. Needs published or explicit Tea coefficients.", reads: [{ label: "Applies to", value: "native Wan, single rank" }] },
  { name: "firstblock / dualblock / dynamicblock", hintLabel: "How", hint: "feature_cache={\"algorithm\": \"firstblock\", \"residual_diff_threshold\": ...}", description: "Block-selective caches. residual_diff_threshold is required; there is no quality-safe default.", reads: [{ label: "Applies to", value: "native Wan, single rank" }] },
  { name: "cfg_gate_step", hintLabel: "Alias", hint: "cfg_gate_fraction", description: "After this fraction of steps, reuse the CFG delta instead of running the negative branch. 1.0 (default) is full CFG.", reads: [{ label: "Rejected with", value: "cfg_parallel=2" }] }
]} />

### Approximate attention (lossy, opt-in)

Never auto-selected. The run is marked `quality_tier="approximate"`. Only the Wan denoiser sets `supports_approximate_attention`.

<KvCatalog heading="Sparse" vars rows={[
  { name: "sta", hintLabel: "How", hint: "approximate_attention={\"kind\": \"sta\", ...}", description: "Sliding-tile attention. Only tuned grids (36, 48, 48) and (18, 48, 80) are accepted. Default Wan2.2 TI2V 121×704×1280 → (31, 22, 40) is rejected before CUDA.", reads: [{ label: "Applies to", value: "native Wan, single rank" }] },
  { name: "vsa", hintLabel: "How", hint: "approximate_attention={\"kind\": \"vsa\", ...}", description: "FastVideo-style video sparse attention. Needs a 3D sparse plan and trained gate_compress VSA-QAT weights. Missing metadata falls back to recorded exact attention.", reads: [{ label: "Applies to", value: "model-specific VSA checkpoints, not generic dense Wan" }] },
  { name: "vmoba", hintLabel: "How", hint: "approximate_attention={\"kind\": \"vmoba\", ...}", description: "Weight-free FastVideo router. Chunk sizes must divide the actual Wan patch grid. The in-tree default profile matches TI2V (31, 22, 40) only. Wrong geometry fails closed.", reads: [{ label: "Needs", value: "fastvideo_kernel MoBA symbols" }] },
  { name: "fastvideo_sla / fastvideo_sagesla", hintLabel: "How", hint: "kind=\"fastvideo_sla\"", description: "Learned Sparse-Linear Attention. Requires a complete proj_l layout in the checkpoint. Official dense Wan and current FastVideo CausalWan Preview weights fail closed. SageSLA also needs spas_sage_attn.", reads: [{ label: "Do not", value: "enable on a dense checkpoint or alias it as LightX2V sla" }] },
  { name: "LightX2V sparse kinds", hintLabel: "Kinds", hint: "dynamic_sparse, sparge, nbhd, lightx2v_sla_mask, flexblock, lightx2v_spas_sage, draft_attn, radial_attn, rainfusion_attn, svg_attn, svg2_attn, lightx2v_svg_mask", description: "Pinned LightX2V training-free sparse lanes. Each call must hit the pinned provider. Ambiguous aliases sla / sagesla are not accepted.", reads: [{ label: "Needs", value: "the pinned LightX2V checkout and a matching operator" }] }
]} />

STA / VSA / VMoBA symbols come from the vendored `thirdparty/fastvideo-kernel` tree (or a matching installed `fastvideo_kernel` wheel). Local build: set `CUDA_HOME` and `CUDACXX`, then `./build.sh` in that directory. ThunderKittens C++ kernels compile only on Hopper `sm_90a` (never on aarch64 hosts); otherwise Triton fallbacks run. FP4 attention / quant CUDA extensions need Blackwell `sm_120a` and CUDA ≥ 12.8. FA4 CuTe block-sparse is an optional `flash_attn.cute` extra, not part of the default build.

### Quantization (experimental)

<KvCatalog heading="Quant" vars rows={[
  { name: "fp8 / nvfp4", hintLabel: "How", hint: "quantization={\"mode\": \"fp8\", \"min_features\": 1024}", description: "Packed low-precision wrappers. Ineligible shapes stay dense and say so. This is a numerical approximation, not a certified quality preset.", reads: [{ label: "Rejected with", value: "block or disk offload" }] },
  { name: "int8", hintLabel: "How", hint: "quantization=\"int8\"", description: "Dynamic per-token / per-channel W8A8 Triton on large GEMMs. Small shapes keep BF16 (counted as dense_policy_calls, not a fallback). Full-pipeline INT8 is opt-in and quality-unsafe by default.", reads: [{ label: "Measured", value: "A full Wan INT8 smoke executed the kernels and still failed latent / video quality (cosine ~0.91, video PSNR ~17). Treat as calibration, not a speed claim." }] },
  { name: "int4", hintLabel: "How", hint: "quantization=\"int4\"", description: "Packed / groupwise storage, then dequantize + dense. Not a claimed INT4 GEMM speedup.", reads: [{ label: "Rejected with", value: "block or disk offload" }] },
  { name: "gguf", hintLabel: "How", hint: "quantization=\"gguf\" or a .gguf file", description: "Real GGUF tensor loader and compressed storage. There is no qualified GGUF low-bit GEMM in this runtime.", reads: [{ label: "Applies to", value: "native loader when the optional gguf package is installed" }] }
]} />

### Preview decode only

`vae_preview_decoder_path` swaps the Wan2.2 48-channel official VAE for TAEW2.2. That is a fast, approximate, decode-only preview. Official VAE tiling, temporal chunking, and VAE-parallel do not apply because the official decoder is not built. Do not use TAEW for final-quality or official-benchmark videos.

## What you cannot do

<KvCatalog heading="Do not" rows={[
  { name: "Count configuration as a speedup", hint: "Not evidence", description: "requested without effective, or pending wrappers, or dense_policy_calls / SDPA fallback, is not acceleration." },
  { name: "Combine sequence parallel with caches, sparse attention, or CUDA Graph", hint: "Rejected at build", description: "SP needs rank-synchronized skip / route decisions that are not implemented. Fused RoPE is the exception — it is SP-aware." },
  { name: "Combine CUDA Graph with feature cache, approximate attention, or static_cross_kv", hint: "Rejected at build", description: "Graph replay would freeze Python cache and sparse schedules from a previous step or request." },
  { name: "Quantization + block or disk offload", hint: "Rejected at build", description: "Quantized weights are buffers; the async two-layer offloader owns Parameters only. Disk offload also cannot materialize the transform." },
  { name: "device_map=balanced + any offload", hint: "Rejected at build", description: "Balanced sharding requires resident VRAM weights." },
  { name: "Two feature caches at once", hint: "Rejected at build", description: "teacache, magcache, adacache, taylorseer, blocktaylorseer, custom, and the block caches are mutually exclusive." },
  { name: "MagCache without ratios, or TeaCache without a preset or coefficients", hint: "Rejected at build", description: "Uncalibrated Wan2.2 MagCache has no in-tree ratios. Unknown TeaCache model IDs must supply coefficients." },
  { name: "STA on the default TI2V 121×704×1280 grid", hint: "Rejected before CUDA", description: "Only (36, 48, 48) and (18, 48, 80) are tuned." },
  { name: "Reuse a VMoBA profile on another resolution or frame count", hint: "Fails closed", description: "Chunk sizes must divide the real post-patch grid. Derive the grid first." },
  { name: "FastVideo SLA on official dense Wan", hint: "Fails closed", description: "Those checkpoints have no proj_l. Do not use LightX2V sla / sagesla aliases for this lane." },
  { name: "sage_attn2_k_int8_v_fp8 as a Wan sparse provider", hint: "Fails closed", description: "That registry entry expects quantized K/V tuples. The Wan generator supplies floating K/V and has no audited per-layer quantize cache." },
  { name: "INT8 / INT4 / GGUF as a default quality path", hint: "Quality cliff", description: "INT8 kernels can run and still wreck latents. INT4 and GGUF are storage plus dense compute unless a future kernel path is certified." },
  { name: "TAEW2.2 as the official VAE", hint: "Preview only", description: "Wrong colors / structure vs the 48-channel Wan2.2 VAE. Keep the official decoder for release and benchmarks." },
  { name: "These Wan knobs on APIs or other families", hint: "Not wired", description: "Wan 2.5+ APIs ignore them. HunyuanVideo takes dtype, offload, attention, vae_tiling. Cosmos Predict2 takes dtype, offload, vae_tiling. Echo-Memory exposes offload_model only. Matrix-Game 3.5 builds a fixed BF16 RuntimePolicy. LTX-2.5 has its own offload_mode / quantization, not this matrix." },
  { name: "PyTorch tricks that are not in RuntimePolicy", hint: "Not wired", description: "Arbitrary torch.compile on a family runtime, sequential offload, xFormers installed but not selected, or a community TeaCache monkey-patch is “possible in PyTorch,” not a WorldFoundry control. Put it here, not under What you can do." },
  { name: "Change official benchmark steps, CFG, size, or enable lossy caches", hint: "Invalid comparison", description: "Scoring belongs in Evaluation. A faster unofficial schedule is a different workload." },
  { name: "Inference accelerators on a training build", hint: "Rejected at build", description: "Training forbids offload, quantization, Sage, TeaCache, CUDA Graph, fuse_qkv, static_cross_kv, and the other inference wrappers." },
  { name: "cfg_parallel degree other than 2", hint: "Rejected at build", description: "The runner only implements degree 1 or 2, and degree 2 needs a live two-rank CFG group." }
]} />

<Callout type="warn" title="OOM and first-run cost">
Block offload is the VRAM escape hatch; resident + INT8 without expandable CUDA segments has OOM’d in the VAE stage because of an extra BF16 copy. `torch.compile` and CUDA Graph make the first request much slower. Do not time that request.
</Callout>

## How to turn knobs on

The same load keys work from Python, CLI, or Workspace.

<Callout type="info" title="Nested objects stay off the CLI">
`feature_cache`, `approximate_attention`, and `quantization` belong in Python kwargs or load JSON — not a guessed CLI flag.
</Callout>

<Tabs items={['Python', 'CLI', 'Workspace']}>
  <Tab>

Complete surface for native Wan 2.1 / 2.2:

```python
from worldfoundry.pipelines.wan.pipeline_wan_2p2 import Wan2p2Pipeline

pipe = Wan2p2Pipeline.from_pretrained(
    model_path="/checkpoints/Wan2.2-TI2V-5B",
    device="cuda",
    # Placement
    offload_mode="resident",
    torch_dtype="bfloat16",
    dit_weight_dtype="bf16",
    # Attention and exact kernels
    attention_backend="flash3",
    fuse_qkv=True,
    fused_rope=True,
    static_cross_kv=True,
    compile=False,
    # Lossy — enable at most one cache, and not with SP / CUDA Graph:
    # feature_cache={"algorithm": "teacache"},
    # Wan VAE decode
    vae_decode_autocast="bf16",
    vae_spatial_tiling=True,
    vae_tile_size=(34, 34),
    vae_tile_stride=(18, 16),
    vae_temporal_chunk_size=4,
)
```

  </Tab>
  <Tab>

Keys on that model’s load schema:

```bash
worldfoundry-eval run wan2.2-ti2v-5b \
  --pipeline.task-profile <task-profile> \
  --pipeline.load.offload-mode resident \
  --pipeline.load.attention-backend flash3 \
  --pipeline.load.fuse-qkv true \
  --pipeline.load.static-cross-kv true \
  --output-dir tmp/worldfoundry_infer/wan2.2-ti2v-5b
```

  </Tab>
  <Tab>

Same job path as [Run inference](/docs/guides/inference):

```bash
python -m worldfoundry.studio.workspace_job infer \
  --model-id wan2.2-ti2v-5b \
  --load-json '{
    "offload_mode": "resident",
    "attention_backend": "flash3",
    "fuse_qkv": true,
    "static_cross_kv": true
  }' \
  --output-dir tmp/worldfoundry_infer/wan2.2-ti2v-5b
```

  </Tab>
</Tabs>

<KvCatalog heading="Studio and multi-GPU" rows={[
  { name: "Workspace UI", hintLabel: "Studio", hint: "checkboxes + dropdown", description: "Compile, CPU offload, fused QKV, static cross-KV, fused RoPE, and attention. Nested objects stay in --load-json." },
  { name: "Multi-GPU", hintLabel: "How", hint: "torchrun + sp_degree", description: "Sequence / CFG / VAE parallel require CUDA and a matching world size." }
]} />

## Confirm what actually ran

```python
dit = pipe.native_pipeline.components.denoiser.runtime_optimization_report()
vae = pipe.native_pipeline.components.decoder.runtime_optimization_report()
print(dit["requested"], dit["effective"], dit["fallbacks"])
print(vae["requested"], vae["effective"], vae["fallbacks"])
```

After one forward:

- **Attention** — `attempts == successes > 0`, no target fallback. Compile: `compiled_graph_traces > 0`.
- **QKV / RoPE / static-KV** — a real call this request. Static-KV: hits, `bypasses=0`.
- **CUDA Graph** — `replay > 0`, and `eager = capture_failed = 0`.
- **VAE tiles / chunks** — last window `> 1`, not lifetime totals.
- **Quantization** — `low_precision_kernel_calls > 0`. `dense_policy_calls` is BF16.

## Next steps

<Cards>
  <Card title="Run inference" description="Environment, CLI, and Workspace commands." href="/docs/guides/inference" />
  <Card title="Models" description="Pick a model ID and its default profile." href="/docs/guides/supported-models" />
  <Card title="Studio" description="Workspace checkboxes and load JSON." href="/docs/guides/studio" />
</Cards>

## Trace stages without synchronizing CUDA

Set `WORLDFOUNDRY_NVTX=1` when recording an Nsight Systems trace. For example,
replace `inference.py` below with your existing inference script:

```bash
WORLDFOUNDRY_NVTX=1 nsys profile --trace=cuda,nvtx,osrt -o inference-trace python inference.py
```

The standard `NativeDiffusionRunner` emits `worldfoundry.run`, `encode`,
`denoise_step`, `denoiser.positive/negative`, and `decode` ranges. The shared
video postprocessor and H.264 writer emit `worldfoundry.postprocess` and
`worldfoundry.output.h264`; packed offload copies emit `worldfoundry.offload.h2d`.
Custom runners only inherit ranges for the shared methods they call.
Ranges label host submission and introduce no CUDA synchronization. They do
not measure client display latency. With the switch unset, ranges do not import
PyTorch or probe CUDA; with no CUDA available, they are a no-op.

## Reusable offload buffers and GQA compatibility

Opt in through the Python helper `enable_layerwise_cpu_offload(..., reuse_buffers=True)` to pack
contiguous weights by dtype into aligned CPU buffers and reuse at most two CUDA
slots. CUDA events prevent overwriting a slot while compute still reads it.
Its report includes `buffer_mode` and `device_buffer_bytes`. Non-contiguous
weights preserve the legacy layout and report `per-parameter`; callers can also
select that path with `reuse_buffers=False` (the default).

The packed path requires `torch.no_grad()` or `torch.inference_mode()`, sequential
layer execution, and unshared parameter storage. It rejects CUDA Graph capture
and state-dict operations while active. Disable the handle before changing
model dtype/device or saving/loading weights. For in-place edits use
`layerwise_offload_mutation_scope(layer)`; `handle.disable()` restores CPU weights
and removes hooks. Packed slots are a memory/copy-management option, not a
promise of higher throughput for every layer size.

The core `scaled_dot_product_attention` uses grouped query heads with zero-stride
K/V views for equal-batch 4D tensors and shared masks when explicitly selecting
`backend="efficient"`, or adapting an older SDPA without `enable_gqa`. Batch- or
head-specific masks retain explicit head expansion. Shape and layout can still
require a reshape copy. Existing native GQA selection remains unchanged.

The local measurements below used synthetic component probes with warmup
excluded, CUDA-event timings, and peak allocated memory. They are not
end-to-end model FPS. The design was
informed by FlashDreams [#637](https://github.com/NVIDIA/flashdreams/pull/637),
[#632](https://github.com/NVIDIA/flashdreams/pull/632), and
[#634](https://github.com/NVIDIA/flashdreams/pull/634).

A local component measurement on 2026-09-18 used one H100 80GB, PyTorch
2.5.1+cu121, FP16, 10 warmups and 50 measured rounds per fresh process:

| Probe | Median CUDA time | Peak allocated memory |
| --- | ---: | ---: |
| Offload, legacy per-parameter copies | 0.932 ms | 37.38 MiB |
| Offload, packed reusable slots | 1.514 ms | 37.38 MiB |
| GQA, explicit K/V head copies | 0.419 ms | 82 MiB |
| GQA, grouped views | 0.358 ms | 19 MiB |

Offload used eight 1024-wide Linear layers, batch 64. GQA used
Q `[1,32,256,64]`, K/V `[1,8,8192,64]`, and the efficient SDPA backend.
Packed offload halved H2D copy calls but was slower for these small layers, so
it stays opt-in. Measure the actual model and its layer sizes before selecting
it. Peak memory includes all live tensors in the process; it is not just weights.