Evaluation

Evaluation turns model outputs into reviewable evidence: prepare inputs, generate or import results, score them, and write a scorecard. Model launch details stay in runners; benchmarks consume normalized GenerationResult rows. Scoring and leaderboard claims are separate — a successful framework import is not the same as an official leaderboard run.

For a first end-to-end path, use Quickstart. For flags and discovery commands, see CLI. For per-benchmark layouts, see Benchmark Hub. For metric recipes, see Metrics.

What to run

• Score a directory with a complete benchmark protocol — worldfoundry-eval score --benchmark <id> --artifacts <dir> ...
• Generate from a dataset manifest, then score metric IDs — worldfoundry-eval generate-score --model <id> --dataset-manifest <file> ...
• Reproduce a known model × benchmark configuration — worldfoundry-eval reproduce --recipe <recipe.yaml> ...
• Run a new/custom model on an existing benchmark — worldfoundry-eval run --model <id> --model-manifest-dir <dir> --benchmark <id> ...
• Score existing JSON/JSONL results — worldfoundry-eval evaluate --mode existing-results ...
• Build a request ledger from a task — worldfoundry-eval task materialize ...
• Run one benchmark × model — worldfoundry-eval run --benchmark <id> --model <id> ...
• Import official-shaped results — worldfoundry-eval zoo benchmark-run --mode official-validation ...
• Try a new benchmark manifest — worldfoundry-eval run --benchmark-manifest-dir <dir> --benchmark <id> ...
• List metric IDs — worldfoundry-eval metric list

Default user shape is worldfoundry-eval run. Use evaluate when you already have materialized requests or results. Use zoo benchmark-run for integrated external benchmarks where you supply the data root, generated artifacts, or official result files.

Choose by intent

The CLI, TUI, and Workspace compile these user intents onto the same two execution requests; they do not maintain separate inference or metric implementations.

• Official reproduction — Pinned recipe, official sample coverage, model generation settings, and benchmark runtime settings. Classification: reproduction.
• Model benchmark — In-tree model plus the benchmark's official prompt/data protocol. Classification: protocol_compliant.
• Score my artifacts — User-provided artifacts plus a benchmark's complete metric protocol. Classification: custom_dataset_metric_evaluation.
• Generate and score my dataset — Dataset manifest plus selected executable metric IDs. Classification: custom_dataset_metric_evaluation.
• Score existing results — WorldFoundry JSON/JSONL results plus selected executable metric IDs. Classification: custom_results_metric_evaluation.

A custom-data score is useful evidence, but is not automatically comparable to an official leaderboard. Likewise, a bounded official component run may prove that the integration works without proving full benchmark comparability. prepared_evaluation.json records the classification, resolved request, configuration sources, and preflight diagnostics for every prepared run; model × benchmark plans also cap their claim at the benchmark catalog's checked-in evidence level.

Models and benchmarks meet only through standard artifacts and manifests. The model runner produces GenerationResult rows; the benchmark adapter maps those artifacts strictly into the official layout; the official runner only executes and parses metrics. Adding a model therefore does not mean copying benchmark code, and adding a benchmark does not mean branching inside every model.

Pipeline

1. Resolve mode, output directory, model/benchmark metadata, metrics, and optional runtime profile.
2. Normalize inputs into GenerationRequest rows (task YAML, dataset manifest, JSON/JSONL, or benchmark samples).
3. Produce GenerationResult rows via a WorldModelRunner or by importing existing results.
4. Score metrics against those rows.
5. Write the audit trail: run manifest, ledgers, metric summaries, report.md, and scorecard.json.

Common commands

# Materialize a small request ledger
worldfoundry-eval task materialize <task-name> \
  --benchmark <benchmark-id> \
  --dataset-manifest /path/to/dataset_manifest.json \
  --num-samples 8 \
  --output-jsonl tmp/requests.jsonl \
  --json

# Score existing results
worldfoundry-eval evaluate \
  --mode existing-results \
  --results-path tmp/results.jsonl \
  --output-dir tmp/evaluate_existing \
  --benchmark-id <benchmark-id> \
  --model-id <model-id> \
  --metric artifact_count \
  --json

# Apply VBench's complete protocol to videos generated elsewhere
worldfoundry-eval score \
  --benchmark vbench \
  --artifacts /path/to/videos \
  --output-dir tmp/vbench_custom \
  --plan-only \
  --json

# Run an in-tree model over a checksum-verified dataset manifest
worldfoundry-eval generate-score \
  --model <model-id> \
  --dataset-manifest /path/to/dataset_manifest.json \
  --metric artifact_count \
  --output-dir tmp/custom_dataset \
  --plan-only \
  --json

# Run one benchmark cell
worldfoundry-eval run \
  --benchmark <benchmark-id> \
  --model <model-id> \
  --output-dir tmp/benchmark_run \
  --mode official-run \
  --json

# Import official-shaped results (framework evidence, not a leaderboard claim)
worldfoundry-eval zoo benchmark-run \
  --benchmark-id <benchmark-id> \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_artifacts> \
  --output-dir tmp/benchmark_zoo/official_validation/<benchmark-id> \
  --json

Remove --plan-only after preflight reports ready: true.

Reproduction recipes

Use --recipe for an immutable model × benchmark cell. A reproduction recipe keeps upstream model and benchmark settings separate, so benchmark parameters cannot silently override orchestrator-owned paths or modes. Scope and eligibility are part of the prepared plan rather than inferred from a successful process exit.

schema_version: worldfoundry-reproduction-recipe-v1
id: paper-table-1-vbench
benchmark:
  id: vbench
model:
  id: <model-id>
  variant: <variant-id>
  parameters:
    num_frames: 81
generation:
  num_inference_steps: 30
  guidance_scale: 7.5
evaluation:
  mode: official-run
  parameters: {}
  env: {}
data:
  requests_path: ./official_requests.jsonl
  id: vbench-official-prompts
reference:
  source: <paper-or-official-config-url>
  reported_score: <value>

Run worldfoundry-eval reproduce --recipe recipe.yaml --output-dir tmp/reproduce --plan-only --json first. The prepared record includes the recipe and requests SHA-256 values, source fields, model/benchmark revisions, scope, and leaderboard candidacy; execution still verifies the benchmark runner and local assets.

Benchmark-specific layouts and flags belong on each Benchmark Hub page.

Outputs

• report.md — Human-readable first look
• scorecard.json — Normalized scores and leaderboard eligibility
• metrics/summary.json — Aggregate metrics for automation
• results.jsonl — Per-sample status, artifacts, errors
• run_manifest.json — Run identity, paths, fingerprint

Modes

• existing-results — Score materialized results; no model inference
• model / generate — Run a model runner, then score
• official-validation — Import official-shaped results through the benchmark runner
• official-run — Run the benchmark runtime declared by the manifest

Use official-validation for framework import evidence. Use official-run only after datasets, checkpoints, environment, and official evaluator requirements are satisfied.

Contracts (short)

• GenerationRequest — Normalized sample input
• WorldModelRunner — Executes generation → GenerationResult
• GenerationResult — Per-sample output + artifacts
• Metric — Turns results into score rows
• scorecard — Final eligibility / comparison record

Subsystem ownership and data flow: Evaluation core. Flag reference: CLI.

Next

• Benchmark Hub — per-benchmark setup and run notes
• Metrics — in-tree scorers and compute APIs
• Supported models — catalog status
• Embodied official runtime — Docker / simulator path