Inference optimization

Which WorldFoundry inference speedups you can turn on, which combinations the runtime rejects, and how to confirm a knob actually ran.

Edit this page GitHub Markdown

Last updated: Oct 7, 2026

On this page

Default recommendation What you can do What you cannot do How to turn knobs on Confirm what actually ran Next steps Trace stages without synchronizing CUDA Reusable offload buffers and GQA compatibility

Experimental

These optimizations are experimental. They may change, may not apply to every model, and should not be treated as a supported production contract.

WorldFoundry only treats a technique as available when a public control exists and the main path consumes it. An accepted flag is not a speedup. After a forward, read requested / effective / fallbacks from runtime_optimization_report().

Pluggable diffusion acceleration covers the Sol-Engine / FlashDreams comparison, native SANA/Wan adapters, removal and request isolation, and configurations accepted or rejected by complete video diagnostics.

Most named knobs below are on the native Wan 2.1 / Wan 2.2 pipelines (Wan2p1*, Wan2p2*, including VACE). Wan 2.5 / 2.6 / 2.7 are hosted APIs and have no local kernel controls. Other native recipes share only a subset (dtype, offload, sometimes attention or VAE tiling). Family runtimes (LTX, CogVideoX, and the rest) keep their own flags — do not copy Wan keys onto them.

Accepted is not effective

Missing kernels, calibration, process groups, or checkpoint weights fail at build time or report an explicit dense / SDPA fallback. Wrapper installation alone (*-wrapper-installed (runtime-pending)) is not acceleration.

Default recommendation

First run vs throughput

First run / official-looking output. Keep the catalog defaults: torch_dtype="bfloat16", offload_mode="resident" when VRAM fits (Studio default for Wan2.2 TI2V-5B), otherwise async-block. Leave feature cache, approximate attention, quantization, CUDA Graph, and TAEW preview off. Do not cut steps or CFG for an official benchmark.

Throughput on the same checkpoint. Stay resident BF16. Set attention_backend to a real Flash / Sage provider for your GPU, then fuse_qkv=True, fused_rope=True, and static_cross_kv=True. Add compile=True only after a warmup request — the first compiled forward is slow. Turn on Wan VAE spatial tiling / temporal chunks if decode OOMs or dominates latency. Add one feature-cache algorithm only if you accept skipped DiT steps.

What you can do

These controls are wired. Enable them from from_pretrained(...), Workspace load JSON, or worldfoundry-eval run --pipeline.load.<kebab-name> when that key is on the model’s load schema.

Precision and placement

Placement

torch_dtype / dit_weight_dtype
Howbfloat16 / bf16
Run the DiT in BF16. FP32 checkpoints convert on load. This is the normal production precision, not an extra experiment.
Applies tonative Wan 2.1 / 2.2; other native recipes that pass dtype into RuntimePolicy
offload_mode=resident
Aliasesnone / fast / resident
Keep weights on GPU. Fastest when the model fits. Studio keeps Wan2.2 TI2V-5B resident on an H100-class card.
Applies tonative visual diffusion loaders that call parse_offload_policy
offload_mode=async-block
Aliasesblock / cpu / layer
Layerwise CPU masters with pinned H2D prefetch (at most two resident layers). Use only when VRAM is tight. The Wan VAE stays resident; only the DiT offloads.
CaveatIncompatible with quantization and with device_map=balanced
offload_mode=component
Howoffload_mode="component"
Whole-module CPU offload. Requires a vram_module_map on that component.
Applies tocomponents that declare a wrapper map
device_map=balanced
Howdevice_map="balanced"
Shard transformer layers across visible GPUs. Requires offload_mode=none so weights stay in VRAM.
Applies tomodules that declare layer_container
inplace_residual
Howinplace_residual=True
In-place residual adds on the Wan DiT to cut activation memory. Inference-only; disabled under autograd.
Applies tonative Wan DiT only

Attention backends

Backend

attention_backend
Valuesauto / flash / flash2 / flash3 / flash4 / sage / sage3 / xformers / sdpa / torch
Dispatcher picks a kernel and reports SDPA if the provider is missing or the shape is ineligible. auto is the safe default.
Applies tonative Wan; HunyuanVideo also accepts attention_backend. Studio dropdown: auto, torch, flash_attn_2, flash_attn_3, sage, xformers
WORLDFOUNDRY_ATTENTION_BACKEND
Envauto / flash / cudnn / efficient / math
Process-wide SDPA policy for runtimes that call install_worldfoundry_inference_infra. Names such as flash_attention_2 map to auto unless you pin a real dispatcher backend on the pipeline.
RelatedWORLDFOUNDRY_ENABLE_TF32 (default on), WORLDFOUNDRY_MATMUL_PRECISION=high, WORLDFOUNDRY_DETERMINISTIC=1

Exact or numerically equivalent Wan kernels

Kernel

fuse_qkv
Howfuse_qkv=True
Merge Q/K/V projections. Optional qkv_strategy=auto|packed|split and qkv_split_threshold (Studio default 8192).
Applies tonative Wan DiT. Confirm fused_blocks and a real projection call after one forward
fused_rope
Howfused_rope=True
Fused 3D RoPE. Safe to combine with Ulysses SP (the kernel takes the rank offset and padding). rope_precision is fp32 or fp64.
Applies tonative Wan DiT
static_cross_kv
Howstatic_cross_kv=True
Reuse text cross-attention K/V for the request. Studio default for Wan2.2 TI2V-5B. Not compatible with CUDA Graph.
Applies tonative Wan DiT

Compile and CUDA Graph

Compile

compile
Howcompile=True or torch_compile=True
torch.compile on the bound forward. Optional compile_mode (default / reduce-overhead / max-autotune / max-autotune-no-cudagraphs), compile_backend, compile_fullgraph, compile_dynamic. First request pays compile + autotune cost.
AlsoWORLDFOUNDRY_TORCH_COMPILE=1 for processes that use compile_module_if_enabled
cuda_graph
Howcuda_graph=True
Capture and replay a stable-shape Wan DiT inner call. Off by default. With Inductor, compile_mode must be max-autotune-no-cudagraphs so graphs are not nested.
Rejected withsequence parallel, feature cache, approximate attention, static_cross_kv, custom Inductor compile_options

Parallelism

Parallel

sequence_parallel / sp_degree
Howtorchrun + sp_degree>=2
Native Ulysses SP: exact attention, shards tokens across GPUs. Requires NCCL before load. Wan2.2 has 24 heads, so degree must divide 24 (2, 3, 4, 6, 8, 12).
Rejected withfeature cache, approximate attention, CUDA Graph
cfg_parallel
Howcfg_parallel=2
Positive and negative CFG branches on two ranks. World size must match the SP×CFG mesh. Degree is only 1 or 2.
Rejected withcfg_gate_step < 1.0
vae_parallel
Howvae_parallel=True or a degree
Rank-parallel Wan VAE tiles with a typed all-gather. Degree must be the full SP×CFG world size when those are on. Needs torchrun WORLD_SIZE>1.
Applies toofficial Wan VAE, not TAEW preview

Wan VAE decode

VAE

vae_decode_autocast
Howvae_decode_autocast="bf16"
Mixed-precision Wan decode. Slight numerical change vs FP32 decode.
Applies tonative Wan VAE
vae_spatial_tiling
Howvae_spatial_tiling=True
Tile spatial decode. Aliases: vae_tiled_decode, or component tiled=True. Default tile 34×34, stride 18×16. A shape that yields one tile is not tiled execution.
Not the same flagvae_tiling is Hunyuan / Cosmos Predict2. Wan ignores that name.
vae_temporal_chunk_size
Howvae_temporal_chunk_size=4
Causal temporal streaming decode. Convolution cache is kept across chunks. A sequence that fits in one chunk is not streaming.
Applies toofficial Wan VAE
vae_channels_last_3d
Howvae_channels_last_3d=True
Convert Conv3d weights to channels-last. Fails if the VAE has no Conv3d.
Applies tonative Wan VAE

HunyuanVideo and Cosmos Predict2 expose their own vae_tiling (and Cosmos tile size/stride) on those pipelines only.

Distilled checkpoints

Wan2.2 public pipelines accept peft_adapter_path and merge a PEFT LoRA after an adapter/base/target audit. Quality and step count belong to that distilled checkpoint. There is no separate “4-step mode” without matching weights.

Feature cache (lossy, one algorithm)

Opt-in, single-rank, mutually exclusive. Each CFG branch gets its own cache. TeaCache has published coefficients for Wan2.1 1.3B / 14B T2V and I2V variants, Wan2.1 Fun / VACE aliases, Wan2.2 Fun / A14B, and Wan2.2 TI2V-5B. Other model IDs must pass coefficients explicitly.

Cache

teacache
Howfeature_cache="teacache" or teacache=True
Skip DiT steps from a residual threshold (default 0.26) plus the model preset. Also teacache=<threshold>.
Applies tonative Wan denoisers with a preset or explicit coefficients
adacache
Howfeature_cache="adacache"
Adaptive residual cache. Same SP / CUDA Graph bans as other caches.
Applies tonative Wan
magcache
Howmagcache={"ratios": [...]}
Requires calibrated per-step ratios. No Wan2.2 preset is published in-tree.
Fails closedmagcache=True without ratios
blocktaylorseer
Howfeature_cache={"algorithm": "blocktaylorseer"}
Pinned LightX2V per-block self / cross / FFN predictor. Default dense pattern [true, false, false, false].
Applies tonative Wan, single rank
taylorseer
Howfeature_cache="taylorseer"
Legacy whole-stack residual predictor. Not LightX2V’s per-block algorithm. Default interval 4.
Applies tonative Wan, single rank
custom
Howfeature_cache={"algorithm": "custom", ...}
LightX2V Tea decision plus first-order stack residual. Needs published or explicit Tea coefficients.
Applies tonative Wan, single rank
firstblock / dualblock / dynamicblock
Howfeature_cache={"algorithm": "firstblock", "residual_diff_threshold": ...}
Block-selective caches. residual_diff_threshold is required; there is no quality-safe default.
Applies tonative Wan, single rank
cfg_gate_step
Aliascfg_gate_fraction
After this fraction of steps, reuse the CFG delta instead of running the negative branch. 1.0 (default) is full CFG.
Rejected withcfg_parallel=2

Approximate attention (lossy, opt-in)

Never auto-selected. The run is marked quality_tier="approximate". Only the Wan denoiser sets supports_approximate_attention.

Sparse

sta
Howapproximate_attention={"kind": "sta", ...}
Sliding-tile attention. Only tuned grids (36, 48, 48) and (18, 48, 80) are accepted. Default Wan2.2 TI2V 121×704×1280 → (31, 22, 40) is rejected before CUDA.
Applies tonative Wan, single rank
vsa
Howapproximate_attention={"kind": "vsa", ...}
FastVideo-style video sparse attention. Needs a 3D sparse plan and trained gate_compress VSA-QAT weights. Missing metadata falls back to recorded exact attention.
Applies tomodel-specific VSA checkpoints, not generic dense Wan
vmoba
Howapproximate_attention={"kind": "vmoba", ...}
Weight-free FastVideo router. Chunk sizes must divide the actual Wan patch grid. The in-tree default profile matches TI2V (31, 22, 40) only. Wrong geometry fails closed.
Needsfastvideo_kernel MoBA symbols
fastvideo_sla / fastvideo_sagesla
Howkind="fastvideo_sla"
Learned Sparse-Linear Attention. Requires a complete proj_l layout in the checkpoint. Official dense Wan and current FastVideo CausalWan Preview weights fail closed. SageSLA also needs spas_sage_attn.
Do notenable on a dense checkpoint or alias it as LightX2V sla
LightX2V sparse kinds
Kindsdynamic_sparse, sparge, nbhd, lightx2v_sla_mask, flexblock, lightx2v_spas_sage, draft_attn, radial_attn, rainfusion_attn, svg_attn, svg2_attn, lightx2v_svg_mask
Pinned LightX2V training-free sparse lanes. Each call must hit the pinned provider. Ambiguous aliases sla / sagesla are not accepted.
Needsthe pinned LightX2V checkout and a matching operator

STA / VSA / VMoBA symbols come from the vendored thirdparty/fastvideo-kernel tree (or a matching installed fastvideo_kernel wheel). Local build: set CUDA_HOME and CUDACXX, then ./build.sh in that directory. ThunderKittens C++ kernels compile only on Hopper sm_90a (never on aarch64 hosts); otherwise Triton fallbacks run. FP4 attention / quant CUDA extensions need Blackwell sm_120a and CUDA ≥ 12.8. FA4 CuTe block-sparse is an optional flash_attn.cute extra, not part of the default build.

Quantization (experimental)

Quant

fp8 / nvfp4
Howquantization={"mode": "fp8", "min_features": 1024}
Packed low-precision wrappers. Ineligible shapes stay dense and say so. This is a numerical approximation, not a certified quality preset.
Rejected withblock or disk offload
int8
Howquantization="int8"
Dynamic per-token / per-channel W8A8 Triton on large GEMMs. Small shapes keep BF16 (counted as dense_policy_calls, not a fallback). Full-pipeline INT8 is opt-in and quality-unsafe by default.
MeasuredA full Wan INT8 smoke executed the kernels and still failed latent / video quality (cosine ~0.91, video PSNR ~17). Treat as calibration, not a speed claim.
int4
Howquantization="int4"
Packed / groupwise storage, then dequantize + dense. Not a claimed INT4 GEMM speedup.
Rejected withblock or disk offload
gguf
Howquantization="gguf" or a .gguf file
Real GGUF tensor loader and compressed storage. There is no qualified GGUF low-bit GEMM in this runtime.
Applies tonative loader when the optional gguf package is installed

Preview decode only

vae_preview_decoder_path swaps the Wan2.2 48-channel official VAE for TAEW2.2. That is a fast, approximate, decode-only preview. Official VAE tiling, temporal chunking, and VAE-parallel do not apply because the official decoder is not built. Do not use TAEW for final-quality or official-benchmark videos.

What you cannot do

Do not

OOM and first-run cost

Block offload is the VRAM escape hatch; resident + INT8 without expandable CUDA segments has OOM’d in the VAE stage because of an extra BF16 copy. torch.compile and CUDA Graph make the first request much slower. Do not time that request.

How to turn knobs on

The same load keys work from Python, CLI, or Workspace.

Nested objects stay off the CLI

feature_cache, approximate_attention, and quantization belong in Python kwargs or load JSON — not a guessed CLI flag.

PythonCLIWorkspace

Complete surface for native Wan 2.1 / 2.2:

from worldfoundry.pipelines.wan.pipeline_wan_2p2 import Wan2p2Pipeline

pipe = Wan2p2Pipeline.from_pretrained(
    model_path="/checkpoints/Wan2.2-TI2V-5B",
    device="cuda",
    # Placement
    offload_mode="resident",
    torch_dtype="bfloat16",
    dit_weight_dtype="bf16",
    # Attention and exact kernels
    attention_backend="flash3",
    fuse_qkv=True,
    fused_rope=True,
    static_cross_kv=True,
    compile=False,
    # Lossy — enable at most one cache, and not with SP / CUDA Graph:
    # feature_cache={"algorithm": "teacache"},
    # Wan VAE decode
    vae_decode_autocast="bf16",
    vae_spatial_tiling=True,
    vae_tile_size=(34, 34),
    vae_tile_stride=(18, 16),
    vae_temporal_chunk_size=4,
)

Keys on that model’s load schema:

worldfoundry-eval run wan2.2-ti2v-5b \
  --pipeline.task-profile <task-profile> \
  --pipeline.load.offload-mode resident \
  --pipeline.load.attention-backend flash3 \
  --pipeline.load.fuse-qkv true \
  --pipeline.load.static-cross-kv true \
  --output-dir tmp/worldfoundry_infer/wan2.2-ti2v-5b

Same job path as Run inference:

python -m worldfoundry.studio.workspace_job infer \
  --model-id wan2.2-ti2v-5b \
  --load-json '{
    "offload_mode": "resident",
    "attention_backend": "flash3",
    "fuse_qkv": true,
    "static_cross_kv": true
  }' \
  --output-dir tmp/worldfoundry_infer/wan2.2-ti2v-5b

Studio and multi-GPU

Workspace UI
Studiocheckboxes + dropdown
Compile, CPU offload, fused QKV, static cross-KV, fused RoPE, and attention. Nested objects stay in --load-json.
Multi-GPU
Howtorchrun + sp_degree
Sequence / CFG / VAE parallel require CUDA and a matching world size.

Confirm what actually ran

dit = pipe.native_pipeline.components.denoiser.runtime_optimization_report()
vae = pipe.native_pipeline.components.decoder.runtime_optimization_report()
print(dit["requested"], dit["effective"], dit["fallbacks"])
print(vae["requested"], vae["effective"], vae["fallbacks"])

After one forward:

Attention — attempts == successes > 0, no target fallback. Compile: compiled_graph_traces > 0.
QKV / RoPE / static-KV — a real call this request. Static-KV: hits, bypasses=0.
CUDA Graph — replay > 0, and eager = capture_failed = 0.
VAE tiles / chunks — last window > 1, not lifetime totals.
Quantization — low_precision_kernel_calls > 0. dense_policy_calls is BF16.

Next steps

Run inference

Environment, CLI, and Workspace commands.

Models

Pick a model ID and its default profile.

Studio

Workspace checkboxes and load JSON.

Trace stages without synchronizing CUDA

Set WORLDFOUNDRY_NVTX=1 when recording an Nsight Systems trace. For example, replace inference.py below with your existing inference script:

WORLDFOUNDRY_NVTX=1 nsys profile --trace=cuda,nvtx,osrt -o inference-trace python inference.py

The standard NativeDiffusionRunner emits worldfoundry.run, encode, denoise_step, denoiser.positive/negative, and decode ranges. The shared video postprocessor and H.264 writer emit worldfoundry.postprocess and worldfoundry.output.h264; packed offload copies emit worldfoundry.offload.h2d. Custom runners only inherit ranges for the shared methods they call. Ranges label host submission and introduce no CUDA synchronization. They do not measure client display latency. With the switch unset, ranges do not import PyTorch or probe CUDA; with no CUDA available, they are a no-op.

Reusable offload buffers and GQA compatibility

Opt in through the Python helper enable_layerwise_cpu_offload(..., reuse_buffers=True) to pack contiguous weights by dtype into aligned CPU buffers and reuse at most two CUDA slots. CUDA events prevent overwriting a slot while compute still reads it. Its report includes buffer_mode and device_buffer_bytes. Non-contiguous weights preserve the legacy layout and report per-parameter; callers can also select that path with reuse_buffers=False (the default).

The packed path requires torch.no_grad() or torch.inference_mode(), sequential layer execution, and unshared parameter storage. It rejects CUDA Graph capture and state-dict operations while active. Disable the handle before changing model dtype/device or saving/loading weights. For in-place edits use layerwise_offload_mutation_scope(layer); handle.disable() restores CPU weights and removes hooks. Packed slots are a memory/copy-management option, not a promise of higher throughput for every layer size.

The core scaled_dot_product_attention uses grouped query heads with zero-stride K/V views for equal-batch 4D tensors and shared masks when explicitly selecting backend="efficient", or adapting an older SDPA without enable_gqa. Batch- or head-specific masks retain explicit head expansion. Shape and layout can still require a reshape copy. Existing native GQA selection remains unchanged.

The local measurements below used synthetic component probes with warmup excluded, CUDA-event timings, and peak allocated memory. They are not end-to-end model FPS. The design was informed by FlashDreams #637, #632, and #634.

A local component measurement on 2026-09-18 used one H100 80GB, PyTorch 2.5.1+cu121, FP16, 10 warmups and 50 measured rounds per fresh process:

Probe	Median CUDA time	Peak allocated memory
Offload, legacy per-parameter copies	0.932 ms	37.38 MiB
Offload, packed reusable slots	1.514 ms	37.38 MiB
GQA, explicit K/V head copies	0.419 ms	82 MiB
GQA, grouped views	0.358 ms	19 MiB

Offload used eight 1024-wide Linear layers, batch 64. GQA used Q [1,32,256,64], K/V [1,8,8192,64], and the efficient SDPA backend. Packed offload halved H2D copy calls but was slower for these small layers, so it stays opt-in. Measure the actual model and its layer sizes before selecting it. Peak memory includes all live tensors in the process; it is not just weights.