---
title: MemoBench
description: "Harvard / MIT visual-memory bench: 360 V-D-R clips, automated metrics plus SAM-3 ORS and VQA. WorldFoundry vendors Step 1; it does not invent a board."
---

MemoBench (Harvard, with MIT and MIT-IBM Watson AI Lab) diagnoses **visual memory** in world generators with a Visible → Disappeared → Reappear paradigm: the target is visible while a physical process runs (V), the camera pans away while the process continues off-screen (D), then the camera returns and the model must regenerate the object in its **updated** state (R). The suite is **360 ground-truth clips** (196 synthetic + 164 real) with camera trajectories and depth.

Scoring has three layers:

- **Step 1 automated** — CLIP-IQA+ / LAION aesthetic, optical-flow smoothness, DINOv2 identity, geometric consistency, camera controllability, ImageReward, GT PSNR/SSIM/LPIPS.
- **ORS** — SAM-3 Object Revisit Score.
- **VQA** — instruction following, object/background, continuity of memory, physics adherence (Gemini or a compatible VLM).

`memobench_average` is the mean over available normalized components. Code: [MemoBench-Team/MemoBench](https://github.com/MemoBench-Team/MemoBench).

WorldFoundry **vendors the runtime** and can run Step 1 over `--generated-synthetic-dir` / `--generated-real-dir`. ORS and VQA stay explicit external stages. Official CSV (`eval_csv`, `ors_scores.csv`, VQA, `leaderboard.csv`) can be imported.

<BenchmarkPageSections id="memobench" locale="en" />

## How to run

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id memobench \
  --mode official-validation \
  --official-results-path <official_results.csv> \
  --generated-artifact-dir <generated_frames> \
  --output-dir tmp/memobench/official-validation \
  --json
```

`--run-official` executes vendored Step 1 (`--mode both|synthetic|real`). V-D-R boundaries rescale to the generated frame count. Env: `WORLDFOUNDRY_MEMOBENCH_RESULTS_PATH`, `WORLDFOUNDRY_MEMOBENCH_GENERATED_SYNTHETIC_DIR`, `WORLDFOUNDRY_MEMOBENCH_GENERATED_REAL_DIR`. Shared stack: `video_quality_dino_motion_stack`.

In-tree pointers: `worldfoundry/data/benchmarks/tasks/external/memobench.yaml`, `worldfoundry/evaluation/tasks/execution/runners/memobench/runtime/memobench`.

Outputs include `scorecard.json` (primary `memobench_average`), `eval_csv`, `ors_scores.csv`, and `leaderboard.csv` when present.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)