WorldFoundry
English中文

RBench

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

RBench (ReVidgen; Peking University DAGroup and ByteDance Seed; ICML 2026) evaluates robot-oriented image-to-video. 650 image-text cases:

  • Task track — 250 cases: common manipulation, long-horizon planning, multi-entity collaboration, spatial relationship, visual reasoning.
  • Embodiment track — 400 cases: dual-arm, humanoid, single-arm, quadruped.

Videos are scored by a VLM judge (GPT or Qwen3-VL) on physical plausibility, task adherence, and stability, plus operator metrics (motion smoothness / amplitude) from GroundingDINO / SAM2 / CoTracker / Q-Align. The paper reports 0.96 Spearman with humans. Companion training data: RoVid-X (4M annotated robot clips). License: CC-BY-4.0.

Paper: RBench. Code: DAGroup-PKU/ReVidgen. Data: DAGroup-PKU/RBench.

WorldFoundry reproduces official eval/4_embodiments and eval/5_tasks aggregation (option-to-score mapping, min-max, PAS and consistency penalties, amplitude filter). It does not run the VLM or the operator stack. rbench_overall is a WorldFoundry composite of the two track scores, not a published RBench number. GPT and Qwen outputs are never mixed.

Official paper teaser: temporal grids that test video models as embodied world models.
Official paper teaser: temporal grids that test video models as embodied world models.

Dimensions

  • Task track
    KindTrack

    250 image-text cases: common manipulation, long-horizon planning, multi-entity collaboration, spatial relationship, visual reasoning.

  • Embodiment track
    KindTrack

    400 cases: dual-arm, humanoid, single-arm, quadruped.

Data composition

  • Size

    PKU / ByteDance Seed robot I2V bench (ICML 2026). 650 image-text cases. VLM + GroundingDINO/SAM2/CoTracker/Q-Align operators. CC-BY-4.0. Paper reports 0.96 Spearman with humans.

  • Sources

    DAGroup-PKU/RBench · DAGroup-PKU/RoVid-X · github.com

  • Splits

    dual_arm · humanoid · single_arm · quad · common_manipulation · long-horizon_planning · multi-entity_collaboration · spatial_relationship · visual_reasoning

  • Modalities

    image · text · video

  • License

    CC-BY-4.0

  • Notes

    WorldFoundry aggregates official eval/4_embodiments and eval/5_tasks trees. rbench_overall is a WorldFoundry composite, not a published RBench number. GPT and Qwen are never mixed.

Metrics

Metric

  • rbench_overall
    DirectionHigher is better

    WorldFoundry composite mean of the two official track scores; not a published RBench number.

    PrimaryRBench Overall

  • embodiment_overall
    DirectionHigher is better

    Task completion and visual quality pooled across the four robot embodiments.

  • task_completion
    DirectionHigher is better

    Mean of normalized physical plausibility and task adherence consistency.

  • visual_quality
    DirectionHigher is better

    Weighted stability and smoothness score after amplitude and consistency penalties.

  • visual_quality_base
    DirectionHigher is better

    0.8 x robot subject stability + 0.2 x motion smoothness before penalties.

  • physical_plausibility
    DirectionHigher is better

    VLM physical plausibility score normalized from the 1-5 scale.

  • task_adherence_consistency
    DirectionHigher is better

    VLM task adherence score normalized from the 1-5 scale.

  • robot_subject_stability
    DirectionHigher is better

    VLM robot and object stability score normalized from the 1-15 option-mapped scale.

  • motion_smoothness
    DirectionHigher is better

    Motion smoothness operator score.

  • motion_amplitude
    DirectionHigher is better

    Robotic-manipulator perceptible amplitude operator score driving the PAS penalty.

  • task_track_overall
    DirectionHigher is better

    ALL_TASKS_MEAN pooled over every scored video across the five task splits.

  • common_manipulation
    DirectionHigher is better

    Normalized rubric score for the common manipulation split.

  • long_horizon_planning
    DirectionHigher is better

    Normalized rubric score for the long-horizon planning split.

  • multi_entity_collaboration
    DirectionHigher is better

    Normalized rubric score for the multi-entity collaboration split.

  • spatial_relationship
    DirectionHigher is better

    Normalized rubric score for the spatial relationship split.

  • visual_reasoning
    DirectionHigher is better

    Normalized rubric score for the visual reasoning split.

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Paper human-alignment table, RBench column. Rethinking Video Generation Model for the Embodied World (2026).

ModelRBench score
Wan 2.50.570
Hailuo v20.565
Veo 30.563
Seedance 1.00.551
Cosmos 2.50.464
LongCat-Video0.437
Wan2.1-14B0.399
CogVideoX-5B0.256

How to run

hf download DAGroup-PKU/RBench --repo-type dataset --local-dir <rbench_data_root>

worldfoundry-eval zoo benchmark-run \
  --benchmark-id rbench \
  --mode official-validation \
  --official-results-path <rbench_results_root> \
  --result-model-id <model_id> \
  --output-dir tmp/rbench/official-validation \
  --json

--track restricts to one track. --vlm-backend selects GPT or Qwen when both trees exist. --generated-artifact-dir adds coverage reporting. Env: WORLDFOUNDRY_RBENCH_RESULTS_PATH, WORLDFOUNDRY_RBENCH_DATA_ROOT, WORLDFOUNDRY_RBENCH_VLM_BACKEND.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/rbench.yaml, worldfoundry/evaluation/tasks/execution/runners/rbench.

Outputs include scorecard.json (primary rbench_overall) and track_breakdown.json.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets