---
title: Evaluation
description: Score generated artifacts with benchmarks, metrics, and reproducible scorecards.
---

Evaluation turns model outputs into reviewable evidence: prepare inputs, generate or import results, score them, and write a scorecard. Model launch details stay in runners; benchmarks consume normalized `GenerationResult` rows. Scoring and leaderboard claims are separate — a successful framework import is not the same as an official leaderboard run.

For a first end-to-end path, use [Quickstart](/docs/quickstart). For flags and discovery commands, see [CLI](/docs/reference/cli). For per-benchmark layouts, see [Benchmark Hub](/docs/evaluation/benchmark-hub). For metric recipes, see [Metrics](/docs/evaluation/metrics).

## What to run

- **Score a directory with a complete benchmark protocol** — `worldfoundry-eval score --benchmark <id> --artifacts <dir> ...`
- **Generate from a dataset manifest, then score metric IDs** — `worldfoundry-eval generate-score --model <id> --dataset-manifest <file> ...`
- **Reproduce a known model × benchmark configuration** — `worldfoundry-eval reproduce --recipe <recipe.yaml> ...`
- **Run a new/custom model on an existing benchmark** — `worldfoundry-eval run --model <id> --model-manifest-dir <dir> --benchmark <id> ...`
- **Score existing JSON/JSONL results** — `worldfoundry-eval evaluate --mode existing-results ...`
- **Build a request ledger from a task** — `worldfoundry-eval task materialize ...`
- **Run one benchmark × model** — `worldfoundry-eval run --benchmark <id> --model <id> ...`
- **Import official-shaped results** — `worldfoundry-eval zoo benchmark-run --mode official-validation ...`
- **Try a new benchmark manifest** — `worldfoundry-eval run --benchmark-manifest-dir <dir> --benchmark <id> ...`
- **List metric IDs** — `worldfoundry-eval metric list`

Default user shape is `worldfoundry-eval run`. Use `evaluate` when you already have materialized requests or results. Use `zoo benchmark-run` for integrated external benchmarks where you supply the data root, generated artifacts, or official result files.

## Choose by intent

The CLI, TUI, and Workspace compile these user intents onto the same two execution requests; they do not maintain separate inference or metric implementations.

- **Official reproduction** — Pinned recipe, official sample coverage, model generation settings, and benchmark runtime settings. Classification: `reproduction`.
- **Model benchmark** — In-tree model plus the benchmark's official prompt/data protocol. Classification: `protocol_compliant`.
- **Score my artifacts** — User-provided artifacts plus a benchmark's complete metric protocol. Classification: `custom_dataset_metric_evaluation`.
- **Generate and score my dataset** — Dataset manifest plus selected executable metric IDs. Classification: `custom_dataset_metric_evaluation`.
- **Score existing results** — WorldFoundry JSON/JSONL results plus selected executable metric IDs. Classification: `custom_results_metric_evaluation`.

A custom-data score is useful evidence, but is not automatically comparable to an official leaderboard. Likewise, a bounded official component run may prove that the integration works without proving full benchmark comparability. `prepared_evaluation.json` records the classification, resolved request, configuration sources, and preflight diagnostics for every prepared run; model × benchmark plans also cap their claim at the benchmark catalog's checked-in evidence level.

Models and benchmarks meet only through standard artifacts and manifests. The model runner produces `GenerationResult` rows; the benchmark adapter maps those artifacts strictly into the official layout; the official runner only executes and parses metrics. Adding a model therefore does not mean copying benchmark code, and adding a benchmark does not mean branching inside every model.

## Pipeline

1. Resolve mode, output directory, model/benchmark metadata, metrics, and optional runtime profile.
2. Normalize inputs into `GenerationRequest` rows (task YAML, dataset manifest, JSON/JSONL, or benchmark samples).
3. Produce `GenerationResult` rows via a `WorldModelRunner` or by importing existing results.
4. Score metrics against those rows.
5. Write the audit trail: run manifest, ledgers, metric summaries, `report.md`, and `scorecard.json`.

## Common commands

```bash
# Materialize a small request ledger
worldfoundry-eval task materialize <task-name> \
  --benchmark <benchmark-id> \
  --dataset-manifest /path/to/dataset_manifest.json \
  --num-samples 8 \
  --output-jsonl tmp/requests.jsonl \
  --json

# Score existing results
worldfoundry-eval evaluate \
  --mode existing-results \
  --results-path tmp/results.jsonl \
  --output-dir tmp/evaluate_existing \
  --benchmark-id <benchmark-id> \
  --model-id <model-id> \
  --metric artifact_count \
  --json

# Apply VBench's complete protocol to videos generated elsewhere
worldfoundry-eval score \
  --benchmark vbench \
  --artifacts /path/to/videos \
  --output-dir tmp/vbench_custom \
  --plan-only \
  --json

# Run an in-tree model over a checksum-verified dataset manifest
worldfoundry-eval generate-score \
  --model <model-id> \
  --dataset-manifest /path/to/dataset_manifest.json \
  --metric artifact_count \
  --output-dir tmp/custom_dataset \
  --plan-only \
  --json

# Run one benchmark cell
worldfoundry-eval run \
  --benchmark <benchmark-id> \
  --model <model-id> \
  --output-dir tmp/benchmark_run \
  --mode official-run \
  --json

# Import official-shaped results (framework evidence, not a leaderboard claim)
worldfoundry-eval zoo benchmark-run \
  --benchmark-id <benchmark-id> \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_artifacts> \
  --output-dir tmp/benchmark_zoo/official_validation/<benchmark-id> \
  --json
```

Remove `--plan-only` after preflight reports `ready: true`.

## Reproduction recipes

Use `--recipe` for an immutable model × benchmark cell. A reproduction recipe keeps upstream model and benchmark settings separate, so benchmark parameters cannot silently override orchestrator-owned paths or modes. Scope and eligibility are part of the prepared plan rather than inferred from a successful process exit.

```yaml
schema_version: worldfoundry-reproduction-recipe-v1
id: paper-table-1-vbench
benchmark:
  id: vbench
model:
  id: <model-id>
  variant: <variant-id>
  parameters:
    num_frames: 81
generation:
  num_inference_steps: 30
  guidance_scale: 7.5
evaluation:
  mode: official-run
  parameters: {}
  env: {}
data:
  requests_path: ./official_requests.jsonl
  id: vbench-official-prompts
reference:
  source: <paper-or-official-config-url>
  reported_score: <value>
```

Run `worldfoundry-eval reproduce --recipe recipe.yaml --output-dir tmp/reproduce --plan-only --json` first. The prepared record includes the recipe and requests SHA-256 values, source fields, model/benchmark revisions, scope, and leaderboard candidacy; execution still verifies the benchmark runner and local assets.

Benchmark-specific layouts and flags belong on each Benchmark Hub page.

## Outputs

- **`report.md`** — Human-readable first look
- **`scorecard.json`** — Normalized scores and leaderboard eligibility
- **`metrics/summary.json`** — Aggregate metrics for automation
- **`results.jsonl`** — Per-sample status, artifacts, errors
- **`run_manifest.json`** — Run identity, paths, fingerprint

## Modes

- **`existing-results`** — Score materialized results; no model inference
- **`model` / `generate`** — Run a model runner, then score
- **`official-validation`** — Import official-shaped results through the benchmark runner
- **`official-run`** — Run the benchmark runtime declared by the manifest

Use `official-validation` for framework import evidence. Use `official-run` only after datasets, checkpoints, environment, and official evaluator requirements are satisfied.

## Contracts (short)

- **`GenerationRequest`** — Normalized sample input
- **`WorldModelRunner`** — Executes generation → `GenerationResult`
- **`GenerationResult`** — Per-sample output + artifacts
- **`Metric`** — Turns results into score rows
- **`scorecard`** — Final eligibility / comparison record

Subsystem ownership and data flow: [Evaluation core](/docs/maintainers/architecture/evaluation-core). Flag reference: [CLI](/docs/reference/cli).

## Next

- [Benchmark Hub](/docs/evaluation/benchmark-hub) — per-benchmark setup and run notes
- [Metrics](/docs/evaluation/metrics) — in-tree scorers and compute APIs
- [Supported models](/docs/guides/supported-models) — catalog status
- [Embodied official runtime](/docs/evaluation/embodied-official-runtime) — Docker / simulator path