Studio internals

Studio is a catalog-driven shell over the same pipeline and artifact contracts used by CLI inference and evaluation. It does not own model weights, scoring logic, or benchmark manifests. Its job is to discover runnable entries, prepare inputs, invoke the right runtime driver, write a RunRecord, and pick preview assets for the UI. The same execution core serves the world realtime frontend, Workspace jobs, conda child processes, and torchrun workers.

Call graph

Studio execution path

Launch

LaunchCLI flagsui/launch_config.py

parse_launch_config()

FrontendFrontend routingui/launcher.py

discover_catalog()

Runtime

DiscoveryCataloginference/catalog.py

StudioManager()

OrchestrationStudioManagerinference/execution.py

prepare_inputs()

PrepareStaged inputsinference/execution.py

runtime_driver_for()

DispatchDriverinference/execution.py

run()

Output

ExecuteRunRecordinference/execution.py

materialize_run()

PreviewPreview assetsinference/execution.py

Keep inference semantics in studio/inference/execution.py and catalog discovery in studio/inference/catalog.py. studio/ui/launcher.py routes standalone frontends, and studio/ui/launch_config.py parses launch options. Page layout, fullscreen, and keyboard controls live in studio/visualization/backends/world.py and world_realtime_client.py; inference semantics belong in the shared execution core.

Files

Catalog discovery

Catalog build is intentionally static. discover_catalog() walks pipeline modules with AST helpers so Studio can list models without importing heavy CUDA stacks or optional dependencies. It merges three sources: pipeline_*.py classes, component-pipeline factories, and action-synthesis classes, then overlays curated overrides and canonical runtime entries (aliases, default kwargs, checkpoints, runtime_kind). Dedup keys are (module_path, class_name). Hidden or abstract ids stay out of the UI list.

Import happens later, only when a run needs a concrete class: StudioManager.import_pipeline_class() loads module_path / class_name from the selected CatalogEntry. That split is why a model can appear in the catalog yet still fail at LOAD if the current environment lacks its runtime imports or local checkpoint.

• Find project root (_project_root() / _project_root_candidates()) — Allows Studio to run from repo root or package context.
• Inspect pipelines (AST helpers around _AstPipelineInfo) — Reads pipeline modules without importing heavy model dependencies.
• Normalize ids (_normalize_model_id()) — Stable lookup key for UI, CLI launch args, and recent runs.
• Build entries (discover_catalog()) — Produces CatalogEntry[] with family, class path, category, backend, aliases, examples.
• Filter entries (filter_catalog()) — Applies search and category filters for the UI list.

Execution core

StudioManager is the orchestration hub. A typical run / stream / init / reset path prepares a request, stages inputs into a run directory, selects a driver from entry.runtime_kind, loads the pipeline (LRU-cached), invokes the driver, then materializes a RunRecord. Drivers exist because not every family is a single __call__: two-stage 3DGS reconstructs then orbits, point-cloud nav reconstructs then renders views, WorldFM needs custom camera and panorama handling. Everything else uses BaseRuntimeDriver.

PreparedInputs is the normalized payload after staging prompts, images, videos, interactions, and runtime kwargs. PipelineContext holds the loaded class, device, variant, and output root. RunRecord is the durable unit the UI reloads: ids, inputs, outputs, artifact paths, preview picks, status, and manifest JSON. Workspace inference may also pass an infer_metadata contract so materialization can enforce required output kinds (video, image, splat, and so on).

• PreparedInputs — Captures staged prompt, images, videos, interactions, runtime kwargs, and metadata.
• RunRecord — Stores run ID, model ID, inputs, outputs, artifacts, preview paths, status, and manifest export.
• PipelineContext — Keeps imported pipeline class, model ref, device, variant, load kwargs, and output root.
• BaseRuntimeDriver — Default driver: load pipeline, run fresh request, optionally initialize/continue streaming, reset state.
• TwoStage3DGSRuntimeDriver — Specialized driver for two-stage 3DGS flows.
• PointCloudNavRuntimeDriver — Specialized driver for point-cloud/navigation previews.
• WorldFMRuntimeDriver — Specialized driver for WorldFM camera pose and panorama workflow.
• StudioManager — Orchestrates import, driver selection, input preparation, run execution, recent-run storage, unload/reset.

Preview and artifact helpers

After the driver returns, materialize_run() turns a heterogeneous result into files under the run directory: metadata JSON, known artifact keys, optional frame-to-video export, mesh conversion, and optional Rerun recordings. When previews are still empty, Studio may scan the output directory depending on WORLDFOUNDRY_STUDIO_ARTIFACT_SCAN_MODE. pick_preview_assets() then ranks artifacts for the UI—preferring generated videos and named previews over staged inputs—and returns the primary video, image, splat/model, gallery, and optional .rrd.

Blank previews usually mean the pipeline wrote files outside expected keys, returned an in-memory object without save(), or produced a video the decoder rejects. Fix the pipeline contract first; only then widen scan mode or conversion helpers.

• export_frames_to_video() — Converts frame sequences into a preview video.
• collect_artifact_paths() — Finds generated files under the output directory.
• pick_preview_assets() — Chooses primary video/image/model/json assets for UI preview.
• maybe_extract_video_preview_image() — Saves a representative image from a video preview.
• convert_model_for_preview() — Converts 3D assets when browser preview needs a different format.
• maybe_build_rerun_rrd() — Builds a Rerun recording when available.
• write_json() — Writes run manifests and input/output records.

UI file boundaries

Standalone browser frontends own launch and presentation. Shared inference modules own model loading, normalized inputs, runtime drivers, and artifact contracts; page and transport handlers call StudioManager.

• Launch parsing — ui/launch_config.py parses CLI flags and environment defaults into StudioLaunchConfig.
• Frontend routing — ui/launcher.py and visualization/backends/frontends.py select world, Viser, Spark, or another frontend from model or artifact metadata.
• Browser UI — visualization/backends/world.py and world_realtime_client.py own layout, fullscreen, keyboard, and joystick controls.
• Realtime transport — visualization/backends/world_realtime.py owns resident sessions, calls StudioManager.run_realtime(), and transports frames and controls over WebRTC/WebSocket.
• Shared execution — inference/execution.py owns models and drivers; realtime interaction transports frames in memory without exporting an MP4 for each control event.

Realtime serving boundaries

The realtime path deliberately separates model state from presentation state:

Call graph

Realtime serving

InputBrowser key edges

resample control edges

ControlRealtimeControlResampler

step resident runtime

ModelResidentWorldRuntime

Extractrealtime_frames_from_result()

session postprocess chain

PostVideoPostprocessStream

encode for transport

TransportResolution / JPEG work

enqueue by policy

Queuelatest-interactive / ordered-quality

WireWebRTC / WebSocket

Models own causal cadence through RealtimeSpec; transports must not infer latent strides or silently change exact segment counts. latest-interactive may evict stale frames after bounded backpressure to protect control latency. ordered-quality never drops and is mandatory for prompt-scheduled or queued segments whose chunk_done frame count is a playback contract.

Post-processors live after decoded RGB extraction, own isolated per-session state, may buffer a chunk, and flush tails in chain order. The default identity processor runs through the same production boundary without changing output, so future VSR, denoise, or interpolation adapters do not enter model-specific inference loops. The RTX adapter preserves layout/container/dtype/range, owns one VFX effect per stream, clones borrowed DLPack output before reuse, and is gated behind the separately licensed rtx_postprocess extra. Every completed chunk records normalized stage timings; use runtime/realtime_regression.py to gate p50/p90, throughput, queue depth, and drops from those traces.

Debug path

Most Studio bugs originate in catalog, input, runtime, or artifact contract mismatches. A missing model is almost always catalog discovery, aliases, or a hidden id. Wrong device or variant usually means launch config and PipelineContext overrides. Inputs that never reach the model fail in prepare_inputs() or driver kwargs binding. Streaming failures often mean init/stream was called without memory state or without a prior reconstruction for two-stage drivers. Recent-run reload failures mean the manifest or artifact paths drifted after materialize_run().

• Model missing from UI — discover_catalog() output, CatalogEntry.aliases, category filter, pipeline file naming.
• Wrong variant/device — StudioLaunchConfig, resolve_cli_variant_id(), PipelineContext.
• Input not reaching model — prepare_inputs(), PreparedInputs, driver run_fresh() kwargs.
• Preview blank — collect_artifact_paths(), pick_preview_assets(), conversion helpers, generated file extension.
• Streaming broken — BaseRuntimeDriver.run_init(), BaseRuntimeDriver.run_continue(), pipeline stream(), memory state.
• Recent run cannot reload — RunRecord.to_manifest(), materialize_run(), load_run(), artifact paths.

For end-user launch and frontend routing, see Studio. For how Studio as a surface compiles user intent into the same requests as CLI and MCP, see Surfaces and orchestration. For the pipeline / operator / GenerationResult contract Studio sits on, see Model runtime.