Evaluation core

Introduction

This page is the evaluation package from the inside: which types stay stable, which function owns a run, and which files a reviewer can trust. Read Workflow first if you only need the path from CLI to disk.

The package flows in one direction: public contracts define JSON-safe shapes, execution runners produce side effects and artifacts, catalog loaders turn YAML into those contracts, and reporting turns metrics into scorecards. API modules should stay import-light; heavy runtime code should never leak backward into evaluation/api/.

For module-level lookup, use the Codebase map.

End-to-end flow

Evaluate and model-only runs enter run_evaluate() in evaluate.py. Model mode generates and aligns results first; both modes then delegate scoring and artifact output to ExistingResultsRunner. Model-plus-benchmark runs enter run_model_benchmark() through execute_prepared_evaluation() and use ManifestBenchmarkRunner for official-suite scoring.

User-facing CLI, TUI, and Workspace flows compile into those request types through execution/orchestration/service.py. The five intent dataclasses, the three planning boundaries (PreparedEvaluation, RunPlan, EvaluateRunRequest / ModelBenchmarkRunRequest), and the compile entrypoints are mapped in Surfaces and orchestration. Frontends must not duplicate generation, artifact materialization, benchmark dispatch, or metric execution; new UX should add an intent-to-request mapping, not another runner. Do not wrap these in a fourth recipe executor.

Prepared evaluations record orthogonal provenance for generation, data, evaluation, runtime, and reference fidelity. Claim level is derived from those fields: a fully pinned official run may be an exact_reproduction, unchanged official data and evaluation may be benchmark_comparable, and custom-data or metric-only runs are diagnostic. For model × benchmark plans, checked-in benchmark evidence is an additional upper bound: a bounded component runner remains executable, but the plan is downgraded to diagnostic until the catalog establishes full official-suite evidence. A caller flag can only narrow this policy, never promote a diagnostic run. score_valid records whether the selected evaluator produced a usable score; leaderboard_valid additionally requires comparable provenance and benchmark-runner evidence.

Public contracts

The evaluation/api/ modules define the shapes shared by runners, CLI, tests, docs exports, and validation. Treat changes here as semver-sensitive: downstream code assumes to_dict() / from_dict() round-trips and stable field names on disk.

At the sample level, a GenerationRequest carries ids, task name, inputs, controls, and expected output schema; a matching GenerationResult carries artifacts, status, errors, timings, and metadata. At the model level, WorldModelRunner is the minimum interface every backend must implement: from_config(), generate(), and cleanup(). At the benchmark level, BenchmarkSpec groups tasks, metrics, splits, tags, and dataset metadata. Metrics implement the Metric interface with compute_sample() and aggregate().

These types are intentionally free of checkpoint paths, subprocess details, or vendor-specific imports. Runners translate catalog YAML into these contracts at run time.

Keyword overrides on a run request replace fields without copying live runners, metrics, or nested configuration objects. The original request remains unchanged; execution and cleanup use the supplied instances.

Run paths

Three run paths cover the common execution shapes. Pick the path that matches your task—do not add conditional branches to the dispatch when a new path will do.

ContractRunner (contract.py) normalizes requests and executes generation with optional cache reuse. It delegates scoring and report output to ExistingResultsRunner, so custom metrics use the same computation and aggregation logic online and offline.

ExistingResultsRunner (existing_results.py) aligns existing results with requests and owns sample ledgers, scorecards, and reports. scoring.py executes bounded metric batches and calls each metric’s aggregate() while preserving direction, raw and normalized statistics, confidence intervals, and skip details. The evaluate facade accepts benchmark metric IDs, built-in offline metric IDs, and metric objects in the same run. Missing requested numeric fields produce invalid metric records with skip reasons.

aggregate_mean(...) accepts Python and NumPy real scalars, using the same numeric eligibility rule for means and valid counts. Scores returned directly and scores restored from checkpoints therefore aggregate identically.

Online and offline paths share result conversion, preserving top-level score fields and shorthand artifact paths. Raw fields are merged into metadata.extra, including fields within score mappings; existing metadata wins same-name conflicts. Each output metric ID belongs to one executor: overlapping declarations are rejected before generation when known, and duplicate dynamically emitted IDs fail scoring. Embodied metric suites use registered directions or explicit defaults for embodied success/reward metrics; unknown field metrics have no assumed direction.

The numeric wildcard expands to the union of fields in successful aligned results before scoring batches begin. Each field then uses the same definition and missing-value handling as an explicit numeric:<field> selection. Expansion is scoped to the run, so the caller can reuse the metric object with another dataset.

run_model_benchmark() (model_benchmark.py:825) chains generation with a benchmark-zoo runner. There is no ModelBenchmarkRunner class. It resolves the benchmark manifest, materializes requests from the task registry, runs the model, materializes artifacts for the external evaluator (ManifestBenchmarkRunner in benchmark_runner.py:870), and hands normalized metrics to reporting. Use it for end-to-end model-plus-benchmark runs.

Both model mode and direct ContractRunner calls hand generated results to the same scoring runner. Python requests expose metric_batch_size (default 32) and resume. Each completed metric batch writes metrics/checkpoint.jsonl; resume compares requests, results, metric implementation, version, and declared parameters / config. New or changed metrics run independently of completed generation. Unversioned legacy callables still execute, but their scores are recomputed on resume. Runner parameters and config participate in generation reuse, and local output files must still exist. Samples that need generation also discard their old metric checkpoints.

Checkpoints record the executor's full metric selection; reports store each output metric's definition separately. Adding an unrelated metric preserves comparability, while changing the components of an aggregate changes its definition. Comparisons reject different definitions of the selected metrics. Winner selection requires valid scores, complete reported metric coverage, and a known direction; partial scores remain visible.

If an executor returns only some of its declared output metrics, missing outputs receive invalid records with metric_not_returned and zero coverage. They remain in the sample denominator and count as skipped. Incomplete bundles are not checkpointed, and older checkpoints missing declared outputs are recomputed on resume.

Embodied closed-loop runs save all planned requests before generation and flush each completed rollout to results.jsonl. Cancellation records an interrupted generation stage; completed results remain available. Shard merging requires every declared shard's results file and a result for every planned request. A completed empty shard is allowed. Across scoring entry points, failed generation contributes zero to generation_success and is skipped by quality metrics.

Reproducible planning

Before generation starts, task-driven runs fix sample IDs, splits, and input keys through RunPlan in plan.py. The plan connects the task registry, optional dataset manifest, and evaluate facade: registry lookup → materialized requests → EvaluateRunRequest. Fingerprinting on the plan makes reruns and cache reuse deterministic across machines.

Materialization in materialize.py turns dataset rows or task samples into GenerationRequest rows. Prove this step with worldfoundry-eval task materialize before spending GPU time—most "works locally, fails in CI" bugs trace to mismatched sample IDs here, not to model loading.

Model resolution

Disk YAML becomes a runnable model through a single front door: resolve_model_zoo_runner() in resolver.py (:386). CLI --model-id flags, Python callers, and Studio previews should all converge here before any checkpoint path is interpreted.

# worldfoundry/data/models/catalog/*.yaml
entry = parse_model_zoo_yaml(path)          # schema.py → ModelZooEntry
entry = ModelZooRegistry().resolve(alias)   # alias lookup
manifest = WorldModelManifest.from_entry(entry)
runner = resolve_model_zoo_runner(manifest) # resolver.py
# → WorldModelRunner instance

Built-in runners register through builtins.py and registry.py; custom targets resolve from catalog runner_target strings. Catalog metadata in manifest.py and schema.py is provenance and readiness—not a substitute for a runnable runner.

The registry creates runners without installing inference settings or wrapping their methods. Configured runners are loaded inside the generation execution context only after resume and cache lookup identify pending samples.

WorldFoundryPipelineRunner retains its construction config and exposes current pipeline target, runtime profile, and generation defaults through parameters. Directly supplied runner objects use these settings for both resume and shared generation-cache lookup.

Benchmark integration

Benchmark metadata follows a parallel catalog path. Task YAML describes per-sample protocol; benchmark-zoo YAML describes integration status, external runners, and metric IDs.

# Benchmark-zoo YAML + task YAML registry
entry = parse_benchmark_zoo_yaml(path)      # → BenchmarkZooEntry
entry = BenchmarkZooRegistry().resolve(id)
spec = BenchmarkSpec.from_catalog(entry, task_registry)
raw = ManifestBenchmarkRunner(spec).run()   # external evaluator
metrics = official_normalizers.normalize(raw)
# → metrics/summary.json + scorecard.json

External benchmarks stay behind the runner boundary: vendor evaluators produce raw output, normalizers convert it into stable metrics, and the shared reporting layer writes metrics/summary.json and scorecard.json.

Metric packages keep declarations importable without optional computation dependencies. Public registry listing includes all built-in package declarations; compute exports use the shared lazy_export helper to load implementations on invocation. An unavailable compute dependency therefore does not hide the metric ID, aliases, or direction.

Reporting chain

Reporting runs after metrics finish. Scorecards are the machine-readable eligibility record; Markdown reports are derived, not authoritative.

per_sample = read_jsonl("metrics/per_sample.jsonl")
summary = aggregate_metrics(per_sample)     # → metrics/summary.json
scorecard = build_scorecard(summary)      # → scorecard.json
write_report_md(scorecard)                # → report.md (derived, not authoritative)

Validation scripts and docs generators import reporting/validation.py and expect stable scorecard fields. Field renames here break release gates. The concrete entry points: build_scorecard at reporting/scorecard.py:136, write_scorecard at :270, build_run_summary at reporting/run_report.py:208, and _validate_scorecard at reporting/validation.py:207.

For how CLI/TUI/MCP compile user intent into the EvaluateRunRequest / ModelBenchmarkRunRequest these runners consume, see Surfaces and orchestration.

Change rules

Use this table before opening a PR that touches evaluation internals. It lists the compatibility surfaces other teams and automation depend on; breaking them requires coordinated updates to validation, docs exports, and example scorecards.

• api/* dataclasses — to_dict() / from_dict() JSON-safe payloads.
• evaluation/tasks/run_* output writing — File names, JSONL row shape, sample_id alignment, final manifest status.
• models/resolver.py — Error messages, runner resolution, model-zoo variant selection.
• evaluation/tasks/official/* schema — Manifest parsing and alias lookup behavior.
• reporting*.py and scorecard.py — Scorecard fields used by validation, docs, and release reports.