WorldFoundry
English中文

MemoBench

Edit this pageGitHubMarkdown

On this page
DimensionsData compositionMetricsLeaderboardHow to run

MemoBench (Harvard, with MIT and MIT-IBM Watson AI Lab) diagnoses visual memory in world generators with a Visible → Disappeared → Reappear paradigm: the target is visible while a physical process runs (V), the camera pans away while the process continues off-screen (D), then the camera returns and the model must regenerate the object in its updated state (R). The suite is 360 ground-truth clips (196 synthetic + 164 real) with camera trajectories and depth.

Scoring has three layers:

  • Step 1 automated — CLIP-IQA+ / LAION aesthetic, optical-flow smoothness, DINOv2 identity, geometric consistency, camera controllability, ImageReward, GT PSNR/SSIM/LPIPS.
  • ORS — SAM-3 Object Revisit Score.
  • VQA — instruction following, object/background, continuity of memory, physics adherence (Gemini or a compatible VLM).

memobench_average is the mean over available normalized components. Code: MemoBench-Team/MemoBench.

WorldFoundry vendors the runtime and can run Step 1 over --generated-synthetic-dir / --generated-real-dir. ORS and VQA stay explicit external stages. Official CSV (eval_csv, ors_scores.csv, VQA, leaderboard.csv) can be imported.

Official paper teaser: Visible–Disappear–Reappear clips that test memory of changing objects.
Official paper teaser: Visible–Disappear–Reappear clips that test memory of changing objects.

Dimensions

  • Synthetic suite
    KindSplit

    196 synthetic V-D-R clips with camera trajectories and depth.

  • Real-world suite
    KindSplit

    164 real-world V-D-R clips.

  • Step 1 automated
    KindTrack

    CLIP-IQA+/LAION, flow, DINOv2, geometry, camera, ImageReward, GT PSNR/SSIM/LPIPS.

  • Object Revisit Score
    KindTrack

    SAM-3 ORS; explicit external stage.

  • VQA pillars
    KindTrack

    Gemini or compatible VLM; instruction, object/background, memory continuity, physics.

Data composition

  • Size

    Harvard / MIT visual-memory bench: 360 ground-truth clips (196 synthetic + 164 real) under Visible → Disappeared → Reappear.

  • Sources

    github.com

  • Splits

    synthetic · real

  • Modalities

    text · image · frames · video

  • License

    MIT

  • Notes

    WorldFoundry vendors Step 1. ORS and VQA stay external. Step 1 only is not the full suite.

Metrics

Metric

  • visual_quality
    DirectionHigher is better

    CLIP-IQA+ and LAION aesthetic quality over the full video.

  • motion_smoothness
    DirectionHigher is better

    Optical-flow temporal coherence over visible and reappear phases.

  • object_identity_consistency
    DirectionHigher is better

    DINOv2 object-centric identity consistency after reappearance.

  • geo3d_consistency
    DirectionHigher is better

    Depth-based geometric consistency across visible and reappear phases.

  • camera_controllability
    DirectionHigher is better

    Alignment between generated and target camera motion.

  • image_reward_score
    DirectionHigher is better

    Human-preference alignment score from the ImageReward path.

  • object_revisit_score
    DirectionHigher is better

    SAM-3 text-prompted detection score for target-object reappearance.

  • gt_all_psnr
    DirectionHigher is better

    PSNR against ground-truth frames across all phases.

  • gt_all_ssim
    DirectionHigher is better

    SSIM against ground-truth frames across all phases.

  • gt_all_lpips
    DirectionLower is better

    LPIPS against ground-truth frames across all phases.

  • instruction_following
    DirectionHigher is better

    VQA pass rate for requested camera movements and events.

  • object_background
    DirectionHigher is better

    VQA pass rate for object identity and background consistency.

  • continuity_of_memory
    DirectionHigher is better

    VQA pass rate for preserving target-object state while out of frame.

  • physics_adherence
    DirectionHigher is better

    VQA pass rate for plausible lighting, shadows, and motion.

  • memobench_average
    DirectionHigher is better

    Mean over available normalized MemoBench component metrics.

    PrimaryMemoBench Average

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. memobench-team.github.io.

memobench-team.github.io · LiveOpen live board

How to run

worldfoundry-eval zoo benchmark-run \
  --benchmark-id memobench \
  --mode official-validation \
  --official-results-path <official_results.csv> \
  --generated-artifact-dir <generated_frames> \
  --output-dir tmp/memobench/official-validation \
  --json

--run-official executes vendored Step 1 (--mode both|synthetic|real). V-D-R boundaries rescale to the generated frame count. Env: WORLDFOUNDRY_MEMOBENCH_RESULTS_PATH, WORLDFOUNDRY_MEMOBENCH_GENERATED_SYNTHETIC_DIR, WORLDFOUNDRY_MEMOBENCH_GENERATED_REAL_DIR. Shared stack: video_quality_dino_motion_stack.

In-tree pointers: worldfoundry/data/benchmarks/tasks/external/memobench.yaml, worldfoundry/evaluation/tasks/execution/runners/memobench/runtime/memobench.

Outputs include scorecard.json (primary memobench_average), eval_csv, ors_scores.csv, and leaderboard.csv when present.

← Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets