Workflow

Introduction

This page traces one evaluation run from the CLI through artifact files on disk. Read it when you need to know where a flag lands in code, which runner owns file writing, or how model mode hands generated results to ExistingResultsRunner. User-facing command examples live in the Evaluation guide; for how CLI/TUI/MCP compile intent into the request this page starts from, see Surfaces and orchestration. Here the focus is execution plumbing.

Entry points

The console entry is worldfoundry/cli/main.py:1888 main(). Most evaluate and model-only runs then enter run_evaluate() in evaluation/tasks/execution/orchestration/evaluate.py (:1071). A model-plus-benchmark run does not go through that facade: orchestration calls execute_prepared_evaluation() → run_model_benchmark() in model_benchmark.py. The CLI dispatches into one of four shapes:

• evaluate — normalizes inputs, picks a run mode, and returns one EvaluateRunResult.
• existing outputs — scores artifacts already on disk through existing_results.py without loading a model.
• model run — resolves a model runner via resolver.py, calls generate(), then scores normalized results.
• model plus benchmark — runs generation and then a benchmark-zoo runner end to end through model_benchmark.py.

For evaluate and model-only runs, the first meaningful branch is mode selection: whether the run needs model weights on GPU, or only files that already exist. Model-plus-benchmark never enters that fork.

Evaluate workflow

The happy path looks like this: one EvaluateRunRequest enters the facade, a mode is chosen, inputs are loaded or materialized from a task registry, and a delegate runner writes the standard artifact bundle. If something is missing from an output directory, follow this flow to see which runner stage should have produced it.

# worldfoundry/cli/main.py → evaluation/tasks/execution/orchestration/evaluate.py
result = run_evaluate(EvaluateRunRequest(...))

mode = _mode(request)
requests, results = load_or_materialize(request)  # task registry / disk paths
runner = pick_delegate_runner(mode, request)

runner.run(requests, results)
# writes under output_dir/:
write("run_manifest.json")
write("execution_plan.json")
write("requests.jsonl")
write("results.jsonl")
write("sample_ledger.jsonl")
write("metrics/summary.json")
write("scorecard.json")
write("summary.json")
write("report.md")

The facade is intentionally small. run_evaluate() validates the request, chooses a mode, then hands off to a runner that owns file writing and per-sample accounting.

Mode behavior

existing-results scores supplied results without loading a generation model. model resolves a runner and generates results first. Both modes resolve metric IDs and objects into the same scoring path.

Model mode detail

The model runner emits GenerationResult objects. After sample alignment, ExistingResultsRunner handles scoring and reporting through scoring.py.

results = runner.generate(requests)
results = _align_model_results(requests, results)
return ExistingResultsRunner().run(
    output_dir=output_dir, requests=requests, results=results, metrics=resolved_metrics,
)

Contract runner detail

run_contract() owns generation and delegates scoring and artifact reporting to ExistingResultsRunner. The output file layout remains shared across direct contract runs, model mode, and offline scoring. Metric progress is persisted in batches; an interrupted run retains metrics/checkpoint.jsonl for resume=True. Shared stage handling records generation, scoring, aggregation, and reporting failures. Starting a new attempt clears the previous finished reports; the run index falls back to the current manifest until a new report is ready. Metric summaries distinguish executed, scored, successful, failed, and skipped samples. A skipped metric value does not become an execution error, and each metric retains its own aggregation policy.

Workspace CLI delegates use the same stage handling. A failed subprocess preserves any fresh partial scores and records its actual exit code; successful normalization does not override execution failure. Reports from an earlier attempt cannot stand in for a missing new scorecard.

# contract.py: generation
results, cache_stats = run_generation_with_cache(requests, runner.generate, ...)
return run_existing_results(
    output_dir=output_dir, requests=requests, results=results, metrics=metrics,
)

# existing_results.py: shared scoring and reporting
for batch in score_samples(...):
    write_sample_progress(batch)
summary = build_metrics_summary(requests, results, per_sample_rows, metrics)
write_scorecard(..., metrics_summary=summary)
write_run_report_artifacts(...)

Generation is the expensive stage: runner exceptions become failed samples, and deterministic outputs can reuse the SQLite generation cache. Reporting stages only run after every sample has an aligned result row.

Artifact contract

These files form the audit trail described in the Evaluation guide. They are stable across run modes: an existing-results run still writes sample_ledger.jsonl and scorecard.json so reviewers can compare it with a full model run on the same benchmark. Paths are relative to the run output directory recorded in run_manifest.json.

Task materialization

Task-driven runs fix sample IDs, splits, and input keys before any model code runs. That makes reruns and cache reuse reproducible: the same task YAML and dataset manifest should yield identical requests.jsonl rows on every machine. The code below connects catalog task YAML to the EvaluateRunRequest that run_evaluate() already understands—see Benchmarks and data for where those YAML files live.

registry = load_task_registry_from_paths(task_yaml_paths)
plan = build_run_plan_from_task_registry(registry, dataset_manifest)
requests = _materialize_requests_from_task(plan)
requests = materialize_generation_requests(requests)
return EvaluateRunRequest(requests=requests, ...)

Use this path when sample IDs, splits, input keys, output keys, and cache policy need to be reproducible before generation starts.