WorldFoundry
English中文

CogVideoX

Method overview: an expert Transformer denoises 3D-causal-VAE video latents, with separate text and vision AdaLN paths.

Zhipu THUDM open T2V/I2V family. 2B T2V, 5B T2V, and 5B I2V routes are runner-verified in-tree.

Edit this pageGitHubMarkdown

CogVideoX is Zhipu AI / THUDM's open video family and still one of the most cited open T2V/I2V baselines. The paper is CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. This catalog card serves three runnable variants through in-tree runtimes: 2B text-to-video, 5B text-to-video, and 5B image-to-video. Runner and demo parity are verified.

Official paper teaser: a collage of CogVideoX-generated video frames from the paper.
Official paper teaser: a collage of CogVideoX-generated video frames from the paper.
Method overview: an expert Transformer denoises 3D-causal-VAE video latents, with separate text and vision AdaLN paths.
Method overview: an expert Transformer denoises 3D-causal-VAE video latents, with separate text and vision AdaLN paths.

The generator is a diffusion Transformer over a 3D causal VAE latent, with an expert Transformer for text (and image) conditioning. T2V exposes num_inference_steps, num_frames, height, width, and fps; I2V adds guidance_scale. The I2V configuration follows the official-style 49-frame, 50-step Diffusers recipe. Weights load from THUDM Hugging Face repos (THUDM/CogVideoX-2b, CogVideoX-5b, CogVideoX-5b-I2V) or a local model_path. License metadata is Apache-2.0 on 2B and other on several 5B cards — review before commercial use. Routes are offline batch only: no streaming or interactive control.

Build your run

Pipeline
CogVideoX2bT2VPipelineworldfoundry.pipelines.cogvideox.pipeline_cogvideox_2b_t2v
TaskInteractive Video interactive-video
Environmentworldfoundry-unified-cu128
DeviceCUDA 12.8
Runnerworldfoundry.pipeline
Loadingfrom_pretrained
Invocationnative_or_standard_invocation
worldfoundry-eval run \
  cogvideox-2b-t2v \
  --pipeline.task-profile interactive-video \
  --json

Variants

  • cogvideox-2b-t2v
    StatusVerified

    text-to-video

    Runtime profilecogvideox-2b-t2v

    Pipeline bindingcogvideox-2b-t2v

  • cogvideox-5b-t2v
    StatusVerified

    text-to-video

    Runtime profilecogvideox-5b-t2v

    Pipeline bindingcogvideox-5b-t2v

  • cogvideox-5b-i2v
    StatusVerified

    image-to-video

    Runtime profilecogvideox-5b-i2v

    Pipeline bindingcogvideox-5b-i2v

What it is for

Use cogvideox-2b-t2v to iterate prompts, cogvideox-5b-t2v when you want the larger T2V card, and cogvideox-5b-i2v to animate a still. Required: prompt (T2V) or prompt + image (I2V). Optional: video. actions is seed. Default artifact on the 5B T2V card: cogvideox-5b-t2v.mp4. Compare clip quality on VBench and compositional prompts on T2V-CompBench. Those are the suites CogVideoX results are usually reported on — not ingested WorldFoundry ranks.

Runtime contract

CogVideoX5bT2VPipeline is the default 5B T2V class; 2B T2V and 5B I2V have sibling pipelines. The expert-conditioning design is the paper's main architectural claim versus a plain T5-conditioned U-Net — at run time that means text (and image) tokens steer the DiT in the 3D VAE latent. THUDM/CogVideoX1.5-5B and CogVideoX1.5-5B-I2V are listed as extra checkpoints, not as the verified Studio routes.

Running in WorldFoundry

Default launch is worldfoundry-eval run cogvideox-2b-t2v in worldfoundry-unified-cu128 (Python 3.11, CUDA 12.8). Swap the run id for cogvideox-5b-t2v or cogvideox-5b-i2v. Runner parity is verified; per-run visual QA stays outside the catalog.