WorldFoundry
English中文

AC3D

Method overview: frozen DiT-XL video blocks with a lightweight trainable camera DiT-XS ControlNet stream.

Snap Research ControlNet plugin for CogVideoX 3D camera control. RealEstate10K demo is runner-verified; dedicated torch 2.4 / CUDA 12.4 env; Snap non-commercial license.

Edit this pageGitHubMarkdown

AC3D is Snap Research's ControlNet-style camera plugin for CogVideoX: text plus a source video and 19-column camera pose rows go in; a camera-steered clip comes out. WorldFoundry vendors the official infer runtime under worldfoundry/synthesis/visual_generation/ac3d/ac3d_runtime and exposes it as AC3DPipeline. Aliases ac3d-cogvideox, ac3d-2b, and ac3d-5b resolve here.

Official paper teaser: AC3D camera-controlled video samples with consistent wide-angle motion.
Official paper teaser: AC3D camera-controlled video samples with consistent wide-angle motion.
Method overview: frozen DiT-XL video blocks with a lightweight trainable camera DiT-XS ControlNet stream.
Method overview: frozen DiT-XL video blocks with a lightweight trainable camera DiT-XS ControlNet stream.

The official paper is AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025, arXiv 2411.18673). Project: snap-research.github.io/ac3d. Code: snap-research/ac3d. License: Snap non-commercial research.

Runner parity is verified: the RealEstate10K camera-control demo ran through the Workspace infer path with local CogVideoX-2B and AC3D ControlNet weights, preserving inference/cli_demo_camera.py. The adapter keeps official 2B ControlNet shape settings even though a 5B ControlNet file is also staged.

Build your run

Pipeline
AC3DPipelineworldfoundry.pipelines.ac3d.pipeline_ac3d
TaskDefault default
Environmentac3d
DeviceCUDA 12.4
Runnerworldfoundry.pipeline
Loadingfrom_pretrained
Invocationnative_or_standard_invocation
worldfoundry-eval run \
  ac3d \
  --pipeline.task-profile default \
  --json

What it is for

Re-render a clip along a prescribed 3D path. Prefer the RealEstate10K-style 2B ControlNet demo.

Launch worldfoundry-eval run ac3d.

Score on CameraBench — directly measures camera-control precision in generated video.

Runtime contract

Required: prompt and video; optional: image; actions: camera_index, camera_index_range, camera_pose_rows; artifacts: ac3d.mp4.

Base weights: THUDM/CogVideoX-2b and THUDM/CogVideoX-5b. ControlNets stage under $WORLDFOUNDRY_CKPT_DIR/ac3d as 2B/checkpoint-10000.pt and 5B/checkpoint-10000.pt. Other pose formats must be converted to the 19-column convention first.

Run in WorldFoundry

Status: Verified. Dedicated environment ac3d (Python 3.10, CUDA 12.4, torch 2.4) — not ABI-compatible with worldfoundry-unified-cu128. Stage CogVideoX base plus the matching ControlNet locally. Run worldfoundry-eval run ac3d. License: Snap non-commercial research. Catalog VRAM is unset.