---
title: Evaluation core
description: How requests, runners, metrics, catalogs, and reporting connect inside the evaluation package.
---

## Introduction

This page is the evaluation package from the inside: which types stay stable, which function owns a run, and which files a reviewer can trust. Read [Workflow](/docs/maintainers/architecture/workflow) first if you only need the path from CLI to disk.

The package flows in one direction: **public contracts** define JSON-safe shapes, **execution runners** produce side effects and artifacts, **catalog loaders** turn YAML into those contracts, and **reporting** turns metrics into scorecards. API modules should stay import-light; heavy runtime code should never leak backward into `evaluation/api/`.

For module-level lookup, use the [Codebase map](/docs/maintainers/architecture/file-index).

## End-to-end flow

Evaluate and model-only runs enter `run_evaluate()` in `evaluate.py`. Model mode generates and aligns results first; both modes then delegate scoring and artifact output to `ExistingResultsRunner`. Model-plus-benchmark runs enter `run_model_benchmark()` through `execute_prepared_evaluation()` and use `ManifestBenchmarkRunner` for official-suite scoring.

<ArchDiagram id="evaluation-core" />

User-facing CLI, TUI, and Workspace flows compile into those request types through `execution/orchestration/service.py`. The five intent dataclasses, the three planning boundaries (`PreparedEvaluation`, `RunPlan`, `EvaluateRunRequest` / `ModelBenchmarkRunRequest`), and the compile entrypoints are mapped in [Surfaces and orchestration](/docs/maintainers/architecture/surfaces-orchestration). Frontends must not duplicate generation, artifact materialization, benchmark dispatch, or metric execution; new UX should add an intent-to-request mapping, not another runner. Do not wrap these in a fourth recipe executor.

Prepared evaluations record orthogonal provenance for generation, data, evaluation, runtime, and reference fidelity. Claim level is derived from those fields: a fully pinned official run may be an `exact_reproduction`, unchanged official data and evaluation may be `benchmark_comparable`, and custom-data or metric-only runs are `diagnostic`. For model × benchmark plans, checked-in benchmark evidence is an additional upper bound: a bounded component runner remains executable, but the plan is downgraded to `diagnostic` until the catalog establishes full official-suite evidence. A caller flag can only narrow this policy, never promote a diagnostic run. `score_valid` records whether the selected evaluator produced a usable score; `leaderboard_valid` additionally requires comparable provenance and benchmark-runner evidence.

## Public contracts

The `evaluation/api/` modules define the shapes shared by runners, CLI, tests, docs exports, and validation. Treat changes here as semver-sensitive: downstream code assumes `to_dict()` / `from_dict()` round-trips and stable field names on disk.

At the sample level, a **`GenerationRequest`** carries ids, task name, inputs, controls, and expected output schema; a matching **`GenerationResult`** carries artifacts, status, errors, timings, and metadata. At the model level, **`WorldModelRunner`** is the minimum interface every backend must implement: `from_config()`, `generate()`, and `cleanup()`. At the benchmark level, **`BenchmarkSpec`** groups tasks, metrics, splits, tags, and dataset metadata. Metrics implement the **`Metric`** interface with `compute_sample()` and `aggregate()`.

These types are intentionally free of checkpoint paths, subprocess details, or vendor-specific imports. Runners translate catalog YAML into these contracts at run time.

Keyword overrides on a run request replace fields without copying live runners, metrics, or nested configuration objects. The original request remains unchanged; execution and cleanup use the supplied instances.

## Run paths

Three run paths cover the common execution shapes. Pick the path that matches your task—do not add conditional branches to the dispatch when a new path will do.

**ContractRunner** (`contract.py`) normalizes requests and executes generation with optional cache reuse. It delegates scoring and report output to `ExistingResultsRunner`, so custom metrics use the same computation and aggregation logic online and offline.

**ExistingResultsRunner** (`existing_results.py`) aligns existing results with requests and owns sample ledgers, scorecards, and reports. `scoring.py` executes bounded metric batches and calls each metric’s `aggregate()` while preserving direction, raw and normalized statistics, confidence intervals, and skip details. The evaluate facade accepts benchmark metric IDs, built-in offline metric IDs, and metric objects in the same run. Missing requested numeric fields produce invalid metric records with skip reasons.

`aggregate_mean(...)` accepts Python and NumPy real scalars, using the same numeric eligibility rule for means and valid counts. Scores returned directly and scores restored from checkpoints therefore aggregate identically.

Online and offline paths share result conversion, preserving top-level score fields and shorthand artifact paths. Raw fields are merged into `metadata.extra`, including fields within score mappings; existing metadata wins same-name conflicts. Each output metric ID belongs to one executor: overlapping declarations are rejected before generation when known, and duplicate dynamically emitted IDs fail scoring. Embodied metric suites use registered directions or explicit defaults for embodied success/reward metrics; unknown field metrics have no assumed direction.

The `numeric` wildcard expands to the union of fields in successful aligned results before scoring batches begin. Each field then uses the same definition and missing-value handling as an explicit `numeric:<field>` selection. Expansion is scoped to the run, so the caller can reuse the metric object with another dataset.

**`run_model_benchmark()`** (`model_benchmark.py:825`) chains generation with a benchmark-zoo runner. There is no `ModelBenchmarkRunner` class. It resolves the benchmark manifest, materializes requests from the task registry, runs the model, materializes artifacts for the external evaluator (`ManifestBenchmarkRunner` in `benchmark_runner.py:870`), and hands normalized metrics to reporting. Use it for end-to-end model-plus-benchmark runs.

Both model mode and direct `ContractRunner` calls hand generated results to the same scoring runner. Python requests expose `metric_batch_size` (default 32) and `resume`. Each completed metric batch writes `metrics/checkpoint.jsonl`; resume compares requests, results, metric implementation, version, and declared `parameters` / `config`. New or changed metrics run independently of completed generation. Unversioned legacy callables still execute, but their scores are recomputed on resume. Runner parameters and config participate in generation reuse, and local output files must still exist. Samples that need generation also discard their old metric checkpoints.

Checkpoints record the executor's full metric selection; reports store each output metric's definition separately. Adding an unrelated metric preserves comparability, while changing the components of an aggregate changes its definition. Comparisons reject different definitions of the selected metrics. Winner selection requires valid scores, complete reported metric coverage, and a known direction; partial scores remain visible.

If an executor returns only some of its declared output metrics, missing outputs receive invalid records with `metric_not_returned` and zero coverage. They remain in the sample denominator and count as skipped. Incomplete bundles are not checkpointed, and older checkpoints missing declared outputs are recomputed on resume.

Embodied closed-loop runs save all planned requests before generation and flush each completed rollout to `results.jsonl`. Cancellation records an `interrupted` generation stage; completed results remain available. Shard merging requires every declared shard's results file and a result for every planned request. A completed empty shard is allowed. Across scoring entry points, failed generation contributes zero to `generation_success` and is skipped by quality metrics.

## Reproducible planning

Before generation starts, task-driven runs fix sample IDs, splits, and input keys through **`RunPlan`** in `plan.py`. The plan connects the task registry, optional dataset manifest, and evaluate facade: registry lookup → materialized requests → `EvaluateRunRequest`. Fingerprinting on the plan makes reruns and cache reuse deterministic across machines.

Materialization in `materialize.py` turns dataset rows or task samples into `GenerationRequest` rows. Prove this step with `worldfoundry-eval task materialize` before spending GPU time—most "works locally, fails in CI" bugs trace to mismatched sample IDs here, not to model loading.

## Model resolution

Disk YAML becomes a runnable model through a single front door: `resolve_model_zoo_runner()` in `resolver.py` (`:386`). CLI `--model-id` flags, Python callers, and Studio previews should all converge here before any checkpoint path is interpreted.

```python
# worldfoundry/data/models/catalog/*.yaml
entry = parse_model_zoo_yaml(path)          # schema.py → ModelZooEntry
entry = ModelZooRegistry().resolve(alias)   # alias lookup
manifest = WorldModelManifest.from_entry(entry)
runner = resolve_model_zoo_runner(manifest) # resolver.py
# → WorldModelRunner instance
```

Built-in runners register through `builtins.py` and `registry.py`; custom targets resolve from catalog `runner_target` strings. Catalog metadata in `manifest.py` and `schema.py` is provenance and readiness—not a substitute for a runnable runner.

The registry creates runners without installing inference settings or wrapping their methods. Configured runners are loaded inside the generation execution context only after resume and cache lookup identify pending samples.

`WorldFoundryPipelineRunner` retains its construction config and exposes current pipeline target, runtime profile, and generation defaults through `parameters`. Directly supplied runner objects use these settings for both resume and shared generation-cache lookup.

## Benchmark integration

Benchmark metadata follows a parallel catalog path. Task YAML describes per-sample protocol; benchmark-zoo YAML describes integration status, external runners, and metric IDs.

```python
# Benchmark-zoo YAML + task YAML registry
entry = parse_benchmark_zoo_yaml(path)      # → BenchmarkZooEntry
entry = BenchmarkZooRegistry().resolve(id)
spec = BenchmarkSpec.from_catalog(entry, task_registry)
raw = ManifestBenchmarkRunner(spec).run()   # external evaluator
metrics = official_normalizers.normalize(raw)
# → metrics/summary.json + scorecard.json
```

External benchmarks stay behind the runner boundary: vendor evaluators produce raw output, normalizers convert it into stable metrics, and the shared reporting layer writes `metrics/summary.json` and `scorecard.json`.

Metric packages keep declarations importable without optional computation dependencies. Public registry listing includes all built-in package declarations; compute exports use the shared `lazy_export` helper to load implementations on invocation. An unavailable compute dependency therefore does not hide the metric ID, aliases, or direction.

## Reporting chain

Reporting runs after metrics finish. Scorecards are the machine-readable eligibility record; Markdown reports are derived, not authoritative.

```python
per_sample = read_jsonl("metrics/per_sample.jsonl")
summary = aggregate_metrics(per_sample)     # → metrics/summary.json
scorecard = build_scorecard(summary)      # → scorecard.json
write_report_md(scorecard)                # → report.md (derived, not authoritative)
```

Validation scripts and docs generators import `reporting/validation.py` and expect stable scorecard fields. Field renames here break release gates. The concrete entry points: `build_scorecard` at `reporting/scorecard.py:136`, `write_scorecard` at `:270`, `build_run_summary` at `reporting/run_report.py:208`, and `_validate_scorecard` at `reporting/validation.py:207`.

For how CLI/TUI/MCP compile user intent into the `EvaluateRunRequest` / `ModelBenchmarkRunRequest` these runners consume, see [Surfaces and orchestration](/docs/maintainers/architecture/surfaces-orchestration).

## Change rules

Use this table before opening a PR that touches evaluation internals. It lists the compatibility surfaces other teams and automation depend on; breaking them requires coordinated updates to validation, docs exports, and example scorecards.

- `api/*` dataclasses — `to_dict()` / `from_dict()` JSON-safe payloads.
- `evaluation/tasks/run_*` output writing — File names, JSONL row shape, `sample_id` alignment, final manifest status.
- `models/resolver.py` — Error messages, runner resolution, model-zoo variant selection.
- `evaluation/tasks/official/*` schema — Manifest parsing and alias lookup behavior.
- `reporting*.py` and `scorecard.py` — Scorecard fields used by validation, docs, and release reports.