CameraBench

CameraBench (Carnegie Mellon) asks whether a video model understands camera motion — pan, tilt, dolly, and the rest of a cinematographer-designed taxonomy, refined by expert annotation. Four families:

• Binary classification — camera_motion_average_precision and camera_motion_roc_auc.
• VQA — yes / no and question accuracy about the camera.
• Retrieval — text, image, and group retrieval of matching motions.
• Caption — caption quality. GPT judging is opt-in (OPENAI_API_KEY, disable with --no-gpt).

camerabench_average is the mean over available official families. In WorldFoundry the same labels can also seed camera-control generation prompts; that is a downstream use, not a hidden leaderboard.

Paper: CameraBench. Project: linzhiqiu.github.io/papers/camerabench. Code: sy77777en/CameraBench, scoring stack linzhiqiu/t2v_metrics. Data: syCen/CameraBench (CC-BY-4.0).

WorldFoundry aggregates prepared score_dir files or imports official JSON. VLM / SfM score generation stays upstream. Shared geometry stack: camera_geometry_stack (VGGT, MoGe, UniDepth, RAFT).

Dimensions

• Binary camera-motion classification

  KindTask

  Average precision and ROC AUC over expert-annotated camera-motion primitives.

• Camera-motion VQA

  KindTask

  Yes/no and question accuracy about camera motion.

• Camera-motion retrieval

  KindTask

  Text, image, and group retrieval of matching camera motions.

• Camera captioning

  KindTask

  Caption quality over camera-motion descriptions. GPT judging is opt-in.

Data composition

• Size

  CMU CameraBench testset (syCen/CameraBench): expert-annotated camera-motion videos with binary, VQA, retrieval, and caption families.

• Sources

  syCen/CameraBench · github.com · github.com

• Modalities

  video · camera · text

• License

  cc-by-4.0

• Notes

  WorldFoundry aggregates prepared score files or imports official JSON. VLM / SfM score generation stays upstream.

Metrics

Metric

• camera_motion_average_precision

  DirectionHigher is better

  CameraBench binary-classification mean average precision.

• camera_motion_roc_auc

  DirectionHigher is better

  CameraBench binary-classification ROC AUC.

• camera_vqa_accuracy

  DirectionHigher is better

  Mean of CameraBench yes/no binary and question accuracy.

• camera_retrieval_accuracy

  DirectionHigher is better

  Mean of CameraBench text, image, and group retrieval scores.

• camera_caption_score

  DirectionHigher is better

  Mean caption score from official CameraBench caption evaluation outputs.

• camerabench_average

  DirectionHigher is better

  Mean over available official CameraBench metric families.

  PrimaryCameraBench Average

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. sy77777en.github.io/CameraBench.

How to run

Import official JSON (strict coverage):

WORLDFOUNDRY_CAMERABENCH_STRICT=1 worldfoundry-eval zoo benchmark-run \
  --benchmark-id camerabench \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --benchmark-data-root <camerabench_data_root> \
  --output-dir tmp/camerabench/official-validation \
  --json

Aggregate a prepared score directory:

WORLDFOUNDRY_CAMERABENCH_STRICT=1 worldfoundry-eval zoo benchmark-run \
  --benchmark-id camerabench \
  --mode official-validation \
  --score-dir <score_dir> \
  --benchmark-data-root <camerabench_data_root> \
  --output-dir tmp/camerabench/score-dir \
  --json

hf download syCen/CameraBench --repo-type dataset \
  --local-dir cache/worldfoundry/data/hfd_datasets/syCen/CameraBench

--task restricts to one family. Env: WORLDFOUNDRY_CAMERABENCH_DATA_ROOT, WORLDFOUNDRY_CAMERABENCH_RESULTS_PATH, WORLDFOUNDRY_CAMERABENCH_SCORE_DIR.

In-tree pointer: worldfoundry/data/benchmarks/tasks/external/camerabench.yaml.

Outputs include scorecard.json (primary camerabench_average), camera_predictions.jsonl when present, and raw_metric_table.jsonl.

Two-stage official protocol

CameraBench splits method-specific scoring from method-agnostic metrics. WorldFoundry is Script 2: it aggregates a prepared score_dir or imports official JSON. Script 1 (Qwen-2.5-VL, MegaSAM, or your own method) stays upstream. Swap only Script 1; keep the JSON contracts below.

Videos used by the official scripts come from syCen/Videos4CameraBnech (note the upstream spelling). A typical data root looks like:

<data_root>/
  binary_classification/
  vqa_and_retrieval/          # one folder per skill
  caption_data.json
  videos/                     # optional local copy of the video set

HF_TOKEN may be required when a Script 1 model downloads weights.

Binary classification

Splits:

• Movement: Move_Up, Move_Down, Move_In, Move_Out, Move_Left, Move_Right
• Rotation: Pan_Left, Pan_Right, Roll_Clockwise, Roll_Counterclockwise, Tilt_Down, Tilt_Up
• Zooming: Zoom_In, Zoom_Out
• Static: Static

Auto-discovery looks for vqa_scores_*.json (or classification_scores_*.json for custom methods). Required record:

{
  "metadata": {
    "method_type": "Your_Method_Name",
    "model_name": "your_model_name",
    "checkpoint": "optional_checkpoint_path",
    "split_name": "Move_Down",
    "generation_timestamp": "2025-01-XX"
  },
  "scores": [
    {
      "sample_id": "0",
      "video_path": "path/to/video.mp4",
      "question": "Is the camera moving downward?",
      "ground_truth_label": "yes",
      "method": "your_method_identifier",
      "score": 0.85,
      "error": null
    }
  ]
}

VQA and retrieval

Skills: complex_description, confusable_motion, has_motion, motion_and_steadiness, motion_direction, motion_speed, only_motion, scene_dynamics, tracking_shot.

Auto-discovery looks for vqa_retrieval_scores_*.json. Official evaluation can run --mode vqa, --mode retrieval, or both. Required record:

{
  "metadata": {
    "method_type": "Your_Method_Name",
    "model_name": "your_model_name",
    "skill_name": "confusable_motion",
    "task_name": "backward_camera_only_vs_backward_ground_only",
    "split_name": "confusable_motion"
  },
  "scores": [
    {
      "sample_id": "0",
      "pos_video": "path/to/positive_video.mp4",
      "neg_video": "path/to/negative_video.mp4",
      "pos_question": "Is the camera moving backward?",
      "neg_question": "Is the ground moving backward?",
      "method": "your_method_identifier",
      "yes_scores": {
        "pos_text_pos_image": 0.85,
        "pos_text_neg_image": 0.23,
        "neg_text_pos_image": 0.31,
        "neg_text_neg_image": 0.78
      },
      "no_scores": {
        "pos_text_pos_image": 0.15,
        "pos_text_neg_image": 0.77,
        "neg_text_pos_image": 0.69,
        "neg_text_neg_image": 0.22
      },
      "error": null
    }
  ]
}

Captioning

Auto-discovery looks for caption_results_*.json. GPT-4o judging needs OPENAI_API_KEY; WorldFoundry --no-gpt skips it. Required record:

{
  "metadata": {
    "method_type": "Your_Method_Name",
    "model_name": "your_model_name"
  },
  "captions": [
    {
      "sample_id": "0",
      "video_path": "path/to/video.mp4",
      "question": "Describe the camera motion in this video.",
      "reference_answer": "The camera pans left while moving forward",
      "method": "your_method_identifier",
      "generated_caption": "The camera is panning to the left and moving forward smoothly",
      "error": null
    }
  ]
}

Catch Script 1 failures and store them as a string in error. Sample counts (total / success / fail) are derived from the file; do not invent them. Unique ids are built from model_name, checkpoint, and split_name.

← Benchmark Hub