报告与证据

Environment evidence、run manifest、scorecard、紧凑 summary 与可读报告的构建 API。

最后更新: 2026年10月7日

本页内容

build_env_requirementsbuild_environmentbuild_run_manifestwrite_run_manifest_artifactsbuild_scorecardwrite_scorecardbuild_run_summarybuild_markdown_reportwrite_run_report_artifacts

Reporting 把执行状态转成持久证据。Builder 返回普通 dictionary，便于组合与测试；writer 则补充规范文件名并写入磁盘。Environment 或 config 进入 manifest 前，疑似 secret 的值会先被脱敏。

本页 symbol 均从 worldfoundry.evaluation.reporting 导入。

`build_env_requirements`

这个 builder 记录所需环境变量名和本地路径是否存在，但不会记录 secret 值。可选依赖组则通过 required_extras 保持显式。

函数

`build_env_requirements`

源码

def build_env_requirements(required_env: Sequence[str | Mapping[str, Any]] = (),required_paths: Sequence[str | Path | Mapping[str, Any]] = (),required_extras: Sequence[str] = (),environ: Mapping[str, str] | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_env_requirements

为 run manifest 收集环境需求声明（软件包、CUDA、资产等）。

参数

required_envSequence[str | Mapping[str, Any]]默认值 ()

Required environment variable names or mappings.

required_pathsSequence[str | Path | Mapping[str, Any]]默认值 ()

Required local paths checked for existence only.

required_extrasSequence[str]默认值 ()

Required optional dependency groups or runtime bundles.

environMapping[str, str] | None默认值 None

Environment mapping used for deterministic tests; defaults to `os.environ`.

返回值

dict[str, Any]

`build_environment`

Environment payload 记录 Python、已安装 package 版本、git metadata 与相关 cache 路径。完整环境清单过大时，可以用 package_names 限定收集范围。

函数

`build_environment`

源码

def build_environment(repo_root: str | Path | None = None,cache_paths: Mapping[str, Any] | None = None,package_names: Sequence[str] | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_environment

快照当前执行环境，写入 run manifest 作为可复现证据。

参数

repo_rootstr | Path | None默认值 None

Repository root used for git metadata.

cache_pathsMapping[str, Any] | None默认值 None

Cache directories or files relevant to the run.

package_namesSequence[str] | None默认值 None

Optional distribution names to record; all installed distributions are recorded when omitted.

返回值

dict[str, Any]

`build_run_manifest`

这个函数会在 runner 的 base manifest 上补充脱敏配置、环境证据、revision、cache path、复现字段与稳定 manifest hash。

函数

`build_run_manifest`

源码

def build_run_manifest(base_manifest: Mapping[str, Any],config: Mapping[str, Any] | None = None,environment: Mapping[str, Any] | None = None,env_requirements: Mapping[str, Any] | None = None,seed: int | str | None = None,cache_paths: Mapping[str, Any] | None = None,environment_path: str | Path | None = None,env_requirements_path: str | Path | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_run_manifest

构建记录请求、环境与产物索引的 run manifest 文档。

参数

base_manifestMapping[str, Any]

Existing runner manifest fields to preserve.

configMapping[str, Any] | None默认值 None

Redacted run configuration snapshot.

environmentMapping[str, Any] | None默认值 None

Environment payload from `build_environment`.

env_requirementsMapping[str, Any] | None默认值 None

Requirement payload from `build_env_requirements`.

seedint | str | None默认值 None

Run seed when the caller has one.

cache_pathsMapping[str, Any] | None默认值 None

Cache path mapping relevant to model, dataset, or runner behavior.

environment_pathstr | Path | None默认值 None

Optional path to the separately written `environment.json`.

env_requirements_pathstr | Path | None默认值 None

Optional path to the separately written `env_requirements.json`.

返回值

dict[str, Any]

`write_run_manifest_artifacts`

自定义 runner 需要一次性写出标准 manifest 三件套时，使用这个 writer。

from worldfoundry.evaluation.reporting import write_run_manifest_artifacts

paths = write_run_manifest_artifacts(
    output_dir="tmp/custom_run",
    base_manifest={
        "run": {"run_id": "custom-0001", "status": "succeeded"},
        "model": {"model_id": "my-model", "revision": "abc123"},
        "dataset": {"dataset_id": "my-samples", "split": "validation"},
    },
    config={"seed": 42, "device": "cuda:0"},
    required_env=("HF_TOKEN",),
    required_paths=("/data/my-samples",),
    required_extras=("video",),
    seed=42,
)

print(paths["run_manifest"])

如果 HF_TOKEN 或 dataset 路径缺失，输出会把它们记录为缺失证据；函数不会暴露 token，也不会虚构路径存在。

函数

`write_run_manifest_artifacts`

源码

def write_run_manifest_artifacts(output_dir: str | Path,base_manifest: Mapping[str, Any],config: Mapping[str, Any] | None = None,required_env: Sequence[str | Mapping[str, Any]] = (),required_paths: Sequence[str | Path | Mapping[str, Any]] = (),required_extras: Sequence[str] = (),seed: int | str | None = None,cache_paths: Mapping[str, Any] | None = None,repo_root: str | Path | None = None,package_names: Sequence[str] | None = None,environ: Mapping[str, str] | None = None,manifest_path: str | Path | None = None,environment_path: str | Path | None = None,env_requirements_path: str | Path | None = None,environment: Mapping[str, Any] | None = None) -> dict[str, Path]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import write_run_manifest_artifacts

把 run manifest JSON（及附属文件）写入运行输出目录。

参数

output_dirstr | Path

Run output directory.

base_manifestMapping[str, Any]

Existing runner manifest fields to enrich.

configMapping[str, Any] | None默认值 None

Run configuration snapshot with secret-like values redacted.

required_envSequence[str | Mapping[str, Any]]默认值 ()

Required environment variable names or mappings.

required_pathsSequence[str | Path | Mapping[str, Any]]默认值 ()

Required local paths verified before execution.

required_extrasSequence[str]默认值 ()

Required optional dependency groups or runtime bundles.

seedint | str | None默认值 None

Run seed when available.

cache_pathsMapping[str, Any] | None默认值 None

Cache path mapping relevant to reproducibility.

repo_rootstr | Path | None默认值 None

Repository root used for git metadata.

package_namesSequence[str] | None默认值 None

Optional distribution names to record.

environMapping[str, str] | None默认值 None

Environment mapping used for deterministic tests.

manifest_pathstr | Path | None默认值 None

Optional destination for `run_manifest.json`.

environment_pathstr | Path | None默认值 None

Optional destination for `environment.json`.

env_requirements_pathstr | Path | None默认值 None

Optional destination for `env_requirements.json`.

environmentMapping[str, Any] | None默认值 None

Optional precomputed environment payload to reuse across related artifacts.

返回值

dict[str, Path]

`build_scorecard`

Scorecard 汇总 run identity、模型、benchmark、dataset、生成、metric、artifact、provenance 与被跳过工作。它的 leaderboard gate 是保守的：metric 成功计算本身不能证明完整 official evidence。

函数

`build_scorecard`

源码

def build_scorecard(run: Mapping[str, Any],benchmark: Mapping[str, Any],model: Mapping[str, Any],dataset: Mapping[str, Any],generation: Mapping[str, Any],metrics_summary: Mapping[str, Any],artifacts: Mapping[str, Any],skipped: Mapping[str, Any] | None = None,leaderboard_evidence: Mapping[str, Any] | None = None,provenance: Mapping[str, Any] | None = None,evaluation_kind: str = 'existing_results',comparison_identity: Mapping[str, Any] | None = None) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_scorecard

从指标聚合与协议结果组装 scorecard 摘要。

参数

runMapping[str, Any]

benchmarkMapping[str, Any]

modelMapping[str, Any]

datasetMapping[str, Any]

generationMapping[str, Any]

metrics_summaryMapping[str, Any]

artifactsMapping[str, Any]

skippedMapping[str, Any] | None默认值 None

leaderboard_evidenceMapping[str, Any] | None默认值 None

provenanceMapping[str, Any] | None默认值 None

evaluation_kindstr默认值 'existing_results'

comparison_identityMapping[str, Any] | None默认值 None

返回值

dict[str, Any]

`write_scorecard`

这个 convenience function 使用相同 keyword arguments 构建 scorecard，再写成 JSON，同时把自身绝对路径记录到 artifacts.scorecard。

函数

`write_scorecard`

源码

def write_scorecard(path: str | Path,**kwargs: Any) -> Path

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import write_scorecard

将 scorecard 产物落盘，供榜单与人工审阅使用。

参数

pathstr | Path

Destination file path for the scorecard JSON.

kwargsAny

Forwarded to :func:build_scorecard.

返回值

PathThe resolved path of the written scorecard file.

`build_run_summary`

Summary 保留 scorecard 中适合比较的子集：identity、fidelity、sample 数、leaderboard 值、eligibility 与 artifact 路径。它适合 run index 和跨 run 比较。

函数

`build_run_summary`

源码

def build_run_summary(scorecard: Mapping[str, Any]) -> dict[str, Any]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_run_summary

构建供报告与 UI 使用的精简运行摘要结构。

参数

scorecardMapping[str, Any]

返回值

dict[str, Any]

`build_markdown_report`

这个 renderer 把紧凑 summary 转成供人阅读的 report.md，不会重新计算分数，也不会改变 eligibility。

函数

`build_markdown_report`

源码

def build_markdown_report(summary: Mapping[str, Any]) -> str

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import build_markdown_report

根据运行摘要与 scorecard 渲染人类可读的 Markdown 报告。

参数

summaryMapping[str, Any]

返回值

str

`write_run_report_artifacts`

输入内存 scorecard 或 scorecard.json 路径后，这个 writer 会同时生成 summary.json 与 report.md。

函数

`write_run_report_artifacts`

源码

def write_run_report_artifacts(output_dir: str | Path,scorecard_path: str | Path | None = None,summary_path: str | Path | None = None,report_path: str | Path | None = None,scorecard: Mapping[str, Any] | None = None) -> dict[str, Path]

worldfoundry.evaluation.reportingfrom worldfoundry.evaluation.reporting import write_run_report_artifacts

把 Markdown/HTML 等报告文件与其他运行证据一并写出。

参数

output_dirstr | Path

Run output directory used for default file paths.

scorecard_pathstr | Path | None默认值 None

Explicit path to the scorecard JSON file.

summary_pathstr | Path | None默认值 None

Explicit destination for the summary JSON file.

report_pathstr | Path | None默认值 None

Explicit destination for the Markdown report file.

scorecardMapping[str, Any] | None默认值 None

In-memory scorecard payload; overrides *scorecard_path*.

返回值

dict[str, Path]A dict mapping `"summary" and "report" to resolved Path` objects.

正常 CLI 和 run_worldfoundry 流程会自动调用这些 writer。只有外部 runtime 或自定义编排层需要输出相同证据 layout 时，才需要直接使用它们。