Reporting

Builders for environment evidence, run manifests, scorecards, compact summaries, and human-readable reports.

Last updated: Oct 7, 2026

On this page

build_env_requirementsbuild_environmentbuild_run_manifestwrite_run_manifest_artifactsbuild_scorecardwrite_scorecardbuild_run_summarybuild_markdown_reportwrite_run_report_artifacts

Reporting converts execution state into durable evidence. Builders return ordinary dictionaries for composition and testing; writer functions add canonical filenames and persist them. Secret-like values are redacted before environment or configuration data enters a manifest.

Import the symbols on this page from worldfoundry.evaluation.reporting.

`build_env_requirements`

This builder records whether required environment variable names and local paths are present without recording secret values. Optional dependency groups remain explicit in required_extras.

function

`build_env_requirements`

Source

def build_env_requirements(required_env: Sequence[str | Mapping[str, Any]] = (),required_paths: Sequence[str | Path | Mapping[str, Any]] = (),required_extras: Sequence[str] = (),environ: Mapping[str, str] | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_env_requirements

Collect environment requirement claims for a run manifest (packages, CUDA, assets).

Parameters

required_envSequence[str | Mapping[str, Any]]default ()

Required environment variable names or mappings.

required_pathsSequence[str | Path | Mapping[str, Any]]default ()

Required local paths checked for existence only.

required_extrasSequence[str]default ()

Required optional dependency groups or runtime bundles.

environMapping[str, str] | Nonedefault None

Environment mapping used for deterministic tests; defaults to `os.environ`.

Returns

dict[str, Any]

`build_environment`

The environment payload captures Python, installed package versions, git metadata, and relevant cache paths. Passing package_names narrows package collection when a full environment inventory would be excessive.

function

`build_environment`

Source

def build_environment(repo_root: str | Path | None = None,cache_paths: Mapping[str, Any] | None = None,package_names: Sequence[str] | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_environment

Snapshot the executing environment for reproducibility evidence in the run manifest.

Parameters

repo_rootstr | Path | Nonedefault None

Repository root used for git metadata.

cache_pathsMapping[str, Any] | Nonedefault None

Cache directories or files relevant to the run.

package_namesSequence[str] | Nonedefault None

Optional distribution names to record; all installed distributions are recorded when omitted.

Returns

dict[str, Any]

`build_run_manifest`

This function enriches a runner's base manifest with redacted configuration, environment evidence, revisions, cache paths, reproducibility fields, and a stable manifest hash.

function

`build_run_manifest`

Source

def build_run_manifest(base_manifest: Mapping[str, Any],config: Mapping[str, Any] | None = None,environment: Mapping[str, Any] | None = None,env_requirements: Mapping[str, Any] | None = None,seed: int | str | None = None,cache_paths: Mapping[str, Any] | None = None,environment_path: str | Path | None = None,env_requirements_path: str | Path | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_run_manifest

Build the run manifest document that records request, environment, and artifact index.

Parameters

base_manifestMapping[str, Any]

Existing runner manifest fields to preserve.

configMapping[str, Any] | Nonedefault None

Redacted run configuration snapshot.

environmentMapping[str, Any] | Nonedefault None

Environment payload from `build_environment`.

env_requirementsMapping[str, Any] | Nonedefault None

Requirement payload from `build_env_requirements`.

seedint | str | Nonedefault None

Run seed when the caller has one.

cache_pathsMapping[str, Any] | Nonedefault None

Cache path mapping relevant to model, dataset, or runner behavior.

environment_pathstr | Path | Nonedefault None

Optional path to the separately written `environment.json`.

env_requirements_pathstr | Path | Nonedefault None

Optional path to the separately written `env_requirements.json`.

Returns

dict[str, Any]

`write_run_manifest_artifacts`

Use the writer when a custom runner should emit the standard manifest trio in one operation.

from worldfoundry.evaluation.reporting import write_run_manifest_artifacts

paths = write_run_manifest_artifacts(
    output_dir="tmp/custom_run",
    base_manifest={
        "run": {"run_id": "custom-0001", "status": "succeeded"},
        "model": {"model_id": "my-model", "revision": "abc123"},
        "dataset": {"dataset_id": "my-samples", "split": "validation"},
    },
    config={"seed": 42, "device": "cuda:0"},
    required_env=("HF_TOKEN",),
    required_paths=("/data/my-samples",),
    required_extras=("video",),
    seed=42,
)

print(paths["run_manifest"])

An absent HF_TOKEN or dataset path is recorded as missing evidence; the function does not reveal the token or invent the path.

function

`write_run_manifest_artifacts`

Source

def write_run_manifest_artifacts(output_dir: str | Path,base_manifest: Mapping[str, Any],config: Mapping[str, Any] | None = None,required_env: Sequence[str | Mapping[str, Any]] = (),required_paths: Sequence[str | Path | Mapping[str, Any]] = (),required_extras: Sequence[str] = (),seed: int | str | None = None,cache_paths: Mapping[str, Any] | None = None,repo_root: str | Path | None = None,package_names: Sequence[str] | None = None,environ: Mapping[str, str] | None = None,manifest_path: str | Path | None = None,environment_path: str | Path | None = None,env_requirements_path: str | Path | None = None,environment: Mapping[str, Any] | None = None) -> dict[str, Path]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import write_run_manifest_artifacts

Persist run manifest JSON (and companions) into the run output directory.

Parameters

output_dirstr | Path

Run output directory.

base_manifestMapping[str, Any]

Existing runner manifest fields to enrich.

configMapping[str, Any] | Nonedefault None

Run configuration snapshot with secret-like values redacted.

required_envSequence[str | Mapping[str, Any]]default ()

Required environment variable names or mappings.

required_pathsSequence[str | Path | Mapping[str, Any]]default ()

Required local paths verified before execution.

required_extrasSequence[str]default ()

Required optional dependency groups or runtime bundles.

seedint | str | Nonedefault None

Run seed when available.

cache_pathsMapping[str, Any] | Nonedefault None

Cache path mapping relevant to reproducibility.

repo_rootstr | Path | Nonedefault None

Repository root used for git metadata.

package_namesSequence[str] | Nonedefault None

Optional distribution names to record.

environMapping[str, str] | Nonedefault None

Environment mapping used for deterministic tests.

manifest_pathstr | Path | Nonedefault None

Optional destination for `run_manifest.json`.

environment_pathstr | Path | Nonedefault None

Optional destination for `environment.json`.

env_requirements_pathstr | Path | Nonedefault None

Optional destination for `env_requirements.json`.

environmentMapping[str, Any] | Nonedefault None

Optional precomputed environment payload to reuse across related artifacts.

Returns

dict[str, Path]

`build_scorecard`

The scorecard combines run identity, model, benchmark, dataset, generation, metrics, artifacts, provenance, and skipped work. Its leaderboard gate is conservative: successful metric computation alone does not establish complete official evidence.

function

`build_scorecard`

Source

def build_scorecard(run: Mapping[str, Any],benchmark: Mapping[str, Any],model: Mapping[str, Any],dataset: Mapping[str, Any],generation: Mapping[str, Any],metrics_summary: Mapping[str, Any],artifacts: Mapping[str, Any],skipped: Mapping[str, Any] | None = None,leaderboard_evidence: Mapping[str, Any] | None = None,provenance: Mapping[str, Any] | None = None,evaluation_kind: str = 'existing_results',comparison_identity: Mapping[str, Any] | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_scorecard

Assemble the scorecard summary from metric aggregates and protocol outcomes.

Parameters

runMapping[str, Any]

benchmarkMapping[str, Any]

modelMapping[str, Any]

datasetMapping[str, Any]

generationMapping[str, Any]

metrics_summaryMapping[str, Any]

artifactsMapping[str, Any]

skippedMapping[str, Any] | Nonedefault None

leaderboard_evidenceMapping[str, Any] | Nonedefault None

provenanceMapping[str, Any] | Nonedefault None

evaluation_kindstrdefault 'existing_results'

comparison_identityMapping[str, Any] | Nonedefault None

Returns

dict[str, Any]

`write_scorecard`

This convenience function builds a scorecard with the same keyword arguments and writes it as JSON. It also indexes its own resolved path under artifacts.scorecard.

function

`write_scorecard`

Source

def write_scorecard(path: str | Path,**kwargs: Any) -> Path

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import write_scorecard

Write scorecard artifacts to disk for leaderboards and human review.

Parameters

pathstr | Path

Destination file path for the scorecard JSON.

kwargsAny

Forwarded to :func:build_scorecard.

Returns

PathThe resolved path of the written scorecard file.

`build_run_summary`

A summary keeps the comparison-oriented subset of a scorecard: identities, fidelity, sample counts, leaderboard values, eligibility, and artifact paths. It is suitable for indexes and cross-run comparisons.

function

`build_run_summary`

Source

def build_run_summary(scorecard: Mapping[str, Any]) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_run_summary

Build a compact run summary structure used by reports and UIs.

Parameters

scorecardMapping[str, Any]

Returns

dict[str, Any]

`build_markdown_report`

This renderer converts a compact summary into the human-readable report.md. It does not recompute scores or change eligibility.

function

`build_markdown_report`

Source

def build_markdown_report(summary: Mapping[str, Any]) -> str

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_markdown_report

Render a human-readable Markdown report from the run summary and scorecard.

Parameters

summaryMapping[str, Any]

Returns

str

`write_run_report_artifacts`

Given an in-memory scorecard or a scorecard.json path, this writer produces summary.json and report.md together.

function

`write_run_report_artifacts`

Source

def write_run_report_artifacts(output_dir: str | Path,scorecard_path: str | Path | None = None,summary_path: str | Path | None = None,report_path: str | Path | None = None,scorecard: Mapping[str, Any] | None = None) -> dict[str, Path]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import write_run_report_artifacts

Write Markdown/HTML (or related) report files beside other run evidence.

Parameters

output_dirstr | Path

Run output directory used for default file paths.

scorecard_pathstr | Path | Nonedefault None

Explicit path to the scorecard JSON file.

summary_pathstr | Path | Nonedefault None

Explicit destination for the summary JSON file.

report_pathstr | Path | Nonedefault None

Explicit destination for the Markdown report file.

scorecardMapping[str, Any] | Nonedefault None

In-memory scorecard payload; overrides *scorecard_path*.

Returns

dict[str, Path]A dict mapping `"summary" and "report" to resolved Path` objects.

In normal CLI and run_worldfoundry flows these writers are called for you. Use them directly when an external runtime or custom orchestration layer must emit the same evidence layout.