WorldFoundry
English中文

MiraBench

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

MiraBench (Tencent ARC Lab) is the evaluation half of MiraData, a long-video dataset with structured captions. It scores long-video generation on six perspectives: temporal consistency, temporal motion strength, 3D consistency, visual quality, text-video alignment, and distribution consistency. The official scorer exposes 17 metrics (DINO/CLIP temporal consistency, RAFT-style motion, 3D reconstruction error, aesthetic / imaging quality, ViCLIP alignment, FVD / FID / KID). The canonical suite is 150 prompts. License: GPL-3.0.

Paper: MiraData. Project: mira-space.github.io. Data: TencentARC/MiraData. Code: mira-space/MiraData.

WorldFoundry vendors the runtime (calculate_score.py) and converts average_score.csv into a scorecard. WORLDFOUNDRY_MIRABENCH_SCORER_BACKEND=mock is wiring only.

Dimensions

  • Temporal consistency / motion
    KindAxis

    DINO/CLIP temporal consistency, RAFT-style motion strength, smoothness.

  • 3D consistency
    KindAxis

    Reconstruction MAE/RMSE; lower is better.

  • Visual quality
    KindAxis

    Aesthetic and imaging quality.

  • Structured caption alignment
    KindAxis

    Camera, object, background, style, overall ViCLIP alignment.

  • Distribution consistency
    KindAxis

    FVD, FID, KID; lower is better.

Data composition

  • Size

    Tencent ARC long-video eval from MiraData. Canonical suite is 150 prompts with structured captions. Official scorer exposes 17 metrics. GPL-3.0.

  • Sources

    github.com

  • Modalities

    text · video

  • License

    GPL-3.0

  • Notes

    WorldFoundry vendors calculate_score.py. Mock backend is wiring only. Complete upstream scorer execution is pending.

Metrics

Metric

  • dynamic_degree
    DirectionHigher is better

    MiraBench RAFT optical-flow dynamic degree for motion strength.

  • tracking_strength
    DirectionHigher is better

    MiraBench tracking-based long-range motion strength.

  • dino_temporal_consistency
    DirectionHigher is better

    MiraBench DINO adjacent-frame structural temporal consistency.

  • clip_temporal_consistency
    DirectionHigher is better

    MiraBench CLIP adjacent-frame semantic temporal consistency.

  • temporal_motion_smoothness
    DirectionHigher is better

    MiraBench AMT-prior temporal motion smoothness.

  • mean_absolute_error
    DirectionLower is better

    MiraBench 3D reconstruction mean absolute error; lower is better.

  • root_mean_square_error
    DirectionLower is better

    MiraBench 3D reconstruction root mean square error; lower is better.

  • aesthetic_quality
    DirectionHigher is better

    MiraBench LAION aesthetic quality score.

  • imaging_quality
    DirectionHigher is better

    MiraBench MUSIQ imaging quality score.

  • camera_alignment
    DirectionHigher is better

    MiraBench ViCLIP camera-caption alignment.

  • main_object_alignment
    DirectionHigher is better

    MiraBench ViCLIP main-object-caption alignment.

  • background_alignment
    DirectionHigher is better

    MiraBench ViCLIP background-caption alignment.

  • style_alignment
    DirectionHigher is better

    MiraBench ViCLIP style-caption alignment.

  • overall_alignment
    DirectionHigher is better

    MiraBench ViCLIP overall text-video alignment.

  • fvd
    DirectionLower is better

    MiraBench Fréchet Video Distance distribution metric; lower is better.

  • fid
    DirectionLower is better

    MiraBench Fréchet Inception Distance distribution metric; lower is better.

  • kid
    DirectionLower is better

    MiraBench Kernel Inception Distance distribution metric; lower is better.

  • mirabench_average
    DirectionHigher is better

    MiraBench official aggregate when supplied by the official scorer or normalized output.

    PrimaryMirabench Average

Leaderboard

Results from the paper (not a live board; not WorldFoundry-run scores). Table 3 TS column. MiraBench has 17 metrics; this is the paper’s motion-tracking score, not a 17-metric average. MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions (2024).

ModelTracking strength (TS)
MiraDiT (MiraData)49.47
MiraDiT (WebVid-10M)22.36
OpenSora16.07
VideoCrafter26.72

How to run

export WORLDFOUNDRY_MIRABENCH_META_CSV=/path/to/mirabench/meta_generated.csv
export WORLDFOUNDRY_MIRABENCH_GT_META_CSV=/path/to/mirabench/meta_gt.csv
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/mirabench/generated_videos
export WORLDFOUNDRY_MIRABENCH_CKPT_PATH=/path/to/mirabench/data/ckpt

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/mirabench/run_mirabench_official_runner.py \
  --run-official \
  --meta-csv "${WORLDFOUNDRY_MIRABENCH_META_CSV}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/mirabench/official-run \
  --strict \
  --json

Import average_score.csv:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id mirabench \
  --mode official-validation \
  --official-results-path /path/to/mirabench/average_score.csv \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/mirabench/official-validation \
  --json

Name videos 1.mp4 … 150.mp4 (from video_idx). Example metas are bundled under worldfoundry/data/benchmarks/assets/mirabench/data/evaluation_example/. --strict fails on missing prompt videos. Full scoring needs AMT, ViCLIP, MUSIQ, FVD/FID/KID, and 3D-consistency assets.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/mirabench.yaml, worldfoundry/evaluation/tasks/execution/runners/mirabench/runtime/mirabench.

Outputs include scorecard.json (primary mirabench_average) and mirabench_average_score.json.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets