Core inference runtime

Run tensor inference in bounded, optionally shape-stable batches.

Only the failed slice is retried. Successful outputs retain input order, and an allocation failure at batch size one is propagated unchanged. `pad_to_batch_size repeats the final real item and trims its outputs; this is intended for independent per-item encoders so Inductor and CUDA Graphs can reuse a small set of static leading dimensions. persistent_forward` declares that a closure is reused across multiple calls, allowing throughput mode to retain one compiled wrapper. It does not enable compilation by itself.

Return a CUDA autocast context and a no-op context for non-CUDA devices.

*device* may be a tensor, `torch.device, or device string. CPU / NPU / missing-torch paths return :func:contextlib.nullcontext`.

Bind correlation fields for the current execution context.

The returned token can be passed to :meth:contextvars.ContextVar.reset, or callers can prefer :func:log_context for scoped use.

Drop cached runtimes and release unreferenced accelerator allocations.

The torch import is intentionally lazy so model discovery remains cheap in processes that never execute a PyTorch policy. `empty_cache` only returns already-unreferenced blocks to the CUDA allocator; clearing the strong references and collecting cycles must happen first.

Compile one module with persistent cache and wrapper reuse when enabled.

Honours `WORLDFOUNDRY_TORCH_COMPILE and the matching _BACKEND / _MODE / _FULLGRAPH / _DYNAMIC / _STRICT env vars when the explicit kwargs are omitted. Already compiled wrappers (_worldfoundry_core_compiled) are returned unchanged. Delegates to :func:worldfoundry.core.execution.compile_cache.compile_module_cached`.

Configure WorldFoundry's unified logging pipeline.

Idempotent: a second call without `force=True` is a no-op, so it is safe to invoke from every framework entry point (CLI, model-runtime worker, …) without duplicating handlers/sinks.

Convert a device string into a `CUDA_VISIBLE_DEVICES` value.

`cuda:N is interpreted as a local index into an inherited CUDA_VISIBLE_DEVICES` list by default, which is the behavior expected by subprocess launchers nested under a scheduler or torchrun process.

Return the accelerator selected for this process, or CPU.

Distributed launchers select a process-local device with `set_device`. Using that current index avoids collapsing every local rank onto CUDA 0.

Return the process-global inference infra state (do not mutate casually).

The returned dataclass is the live singleton updated by :func:install_worldfoundry_inference_infra and :func:uninstall_worldfoundry_inference_infra.

Install WorldFoundry core inference optimizations for this process.

Environment controls: - `WORLDFOUNDRY_USE_CORE_INFRA=0 disables installation. - WORLDFOUNDRY_ATTENTION_BACKEND=auto|flash|cudnn|efficient|math selects the SDPA backend policy. Attention dispatch backend names (for example flash_attention_2, sage_attention, xformers) are also accepted; they resolve the SDPA policy to auto and are honoured by worldfoundry.core.attention instead. - WORLDFOUNDRY_MATMUL_PRECISION=highest|high|medium selects PyTorch float32 matmul precision. - WORLDFOUNDRY_ENABLE_TF32=0 disables TF32 backend flags. - WORLDFOUNDRY_PATCH_SDPA=0 avoids monkey-patching PyTorch SDPA calls. - WORLDFOUNDRY_DETERMINISTIC=1 enables torch.use_deterministic_algorithms` and CuDNN deterministic mode (XC-23).

Use :func:uninstall_worldfoundry_inference_infra (or the :func:worldfoundry_inference_infra_disabled context manager) to restore the original SDPA function and pre-install torch backend flags.

Serialize current correlation fields for a framework-owned child process.

This deliberately carries only structured context, never logger handlers or raw stdout/stderr. It is safe to merge into a child `env` mapping.

Read a bounded UTF-8 tail from a potentially very large log file.

Seeks to `max(size - max_bytes, 0) and keeps the last *max_lines* decoded lines. A mid-file seek drops the first partial line so the result does not start with a torn UTF-8 sequence. Missing files yield ""`.

Choose a capability-neutral inference batch size from free device memory.

The default is treated as the batch size for roughly 24 GiB of free memory. Smaller devices start conservatively, while 80 GiB and newer high-memory accelerators can use larger batches. Scoped and global environment values take precedence over automatic sizing.

Resolve a concrete inference device without silently selecting the wrong GPU.

Bare `cuda resolves to the CUDA device currently selected for this process. Explicit indices are preserved, which is important when a caller deliberately selects (for example) cuda:4` under an eight-GPU workspace.

Resolve an inference dtype using the selected accelerator's capability.

`auto` selects bf16 on Ampere/Hopper/Blackwell-or-newer CUDA devices, fp16 on older CUDA devices, and fp32 on CPU. Explicit unsupported bf16 is rejected in strict mode instead of producing a later kernel failure.

Run a subprocess with bounded log memory and timeout-safe descendants.

The child receives the current WorldFoundry correlation context and gets a dedicated JSONL lifecycle artifact beside its raw stdout/stderr captures. This keeps framework-owned workers observable without requiring changes to vendored official benchmark code.

Run a single-node Python module under torchrun and capture its logs.

Uses :func:subprocess.run with `capture_output=True and check=False. Prefer :func:run_logged_subprocess` when the worker may run long enough that in-memory capture is unsafe.

Terminate a process and all descendants in its dedicated process group.

POSIX path sends `SIGTERM to the group (os.killpg), waits *grace_seconds*, then SIGKILL. Windows falls back to terminate / kill on the parent only. Already-exited groups are a no-op. The child must have been started with start_new_session=True (see :func:run_logged_subprocess`).

Terminate one process and recursively terminate descendants without a process group.

Used when the child was *not* started in a new session. With `psutil installed, children are terminate'd first, then the parent, then any survivors are kill`'d. Without psutil only the parent is signalled — grandchildren may leak.

Build a single-node torchrun command for a Python module.

Equivalent to `python -m torch.distributed.run --standalone --nproc_per_node N -m MODULE ...`. Does not launch the process.

Run model inference under the shared WorldFoundry core runtime policy.

Installs process-wide infra if needed, then yields inside `torch.no_grad()` when torch is importable. A missing torch install still enters the context so CPU-only control planes do not crash.

Wrap a runner instance so `generate` always uses core inference infra.

Installs infra, then replaces `runner.generate with a wrapper that enters :func:worldfoundry_inference_context. Idempotent: a second call returns the same instance. Runners without generate` are returned unchanged.

Append one canonical lifecycle event to a run-owned JSONL artifact.

Use this for process boundaries where a child may not import the central logger. Normal in-process application events should use :meth:WorldFoundryLoggerAdapter.event instead.