Run 与 benchmark

Existing artifact、模型推理、单个 model × benchmark cell 和 suite 的统一进程内调度。

最后更新: 2026年10月7日

本页内容

一个完整的无 GPU run WorldFoundryRunRequestWorldFoundryRunResultrun_worldfoundrylist_video_benchmarksrun_benchmarknormalize_upstream_resultsbenchmark_integration_spec

run_worldfoundry 是宽入口。它检查一个 typed request，再分发到 existing-results、模型生成、单个 model × benchmark 或 suite runner。若生成 artifact 已经存在，只需要运行 official evaluator 或 normalizer，则应使用更窄的 benchmark facade。

本页 symbol 均从 worldfoundry.evaluation.public 导入。

一个完整的无 GPU run

这个例子创建一份很小的 trajectory artifact，写出 request/result ledger，再用内置 artifact_count metric 评测已有结果。它会经过真实 run 与 reporting 路径，但不会加载模型。

from pathlib import Path

from worldfoundry.evaluation.api import ArtifactRef, GenerationRequest, GenerationResult
from worldfoundry.evaluation.public import WorldFoundryRunRequest, run_worldfoundry

root = Path("tmp/python_api_example")
trace_path = root / "artifacts" / "trajectory.json"
trace_path.parent.mkdir(parents=True, exist_ok=True)
trace_path.write_text('{"actions":["forward","left"]}\n', encoding="utf-8")

request = GenerationRequest(sample_id="nav-0001", task_name="navigation-trace")
result = GenerationResult(
    sample_id=request.sample_id,
    model_id="existing-trace",
    artifacts={
        "trajectory": ArtifactRef.from_path(trace_path, kind="trajectory"),
    },
)

requests_path = root / "requests.jsonl"
results_path = root / "results.jsonl"
requests_path.write_text(request.to_json() + "\n", encoding="utf-8")
results_path.write_text(result.to_json() + "\n", encoding="utf-8")

outcome = run_worldfoundry(
    WorldFoundryRunRequest(
        output_dir=root / "evaluation",
        requests_path=requests_path,
        results_path=results_path,
        metrics=("artifact_count",),
    )
)

assert outcome.ok
print(outcome.to_dict()["scorecard_path"])

输出目录包含对齐后的 ledger、execution plan、metric row、run_manifest.json、summary.json、report.md 和 scorecard.json。这个 run 可以证明 existing-results artifact check 成功，但不会因此变成 official benchmark 或 leaderboard 结果。

`WorldFoundryRunRequest`

这个 request 刻意覆盖多个 mode。results_path 选择 existing-results；只有 model ID 时选择模型执行；model 与 benchmark ID 同时存在时选择一个 benchmark cell；多个选择或 suite ID 会进入矩阵 runner。execute=False 可以先规划兼容 cell，不立即消耗计算资源。

类

`WorldFoundryRunRequest`

源码

class WorldFoundryRunRequest(output_dir: str | Path,model_ids: Sequence[str] = (),benchmark_ids: Sequence[str] = (),suite_ids: Sequence[str] = (),all_benchmarks: bool = False,benchmark_id: str | None = None,benchmark_manifest_dir: str | Path = BENCHMARK_ZOO_DIR,model_manifest_dir: str | Path | None = MODEL_ZOO_DIR,suite_preset_path: str | Path | None = None,engine: str = 'auto',benchmark_mode: str = 'official-run',execute: bool = True,model_workers: int = 1,worker_cuda_devices: Sequence[str] = (),resume: bool = False,skip_incompatible: bool = True,fail_on_skipped: bool = False,model_runner: str | None = None,model_variant_id: str | None = None,model_parameters: Mapping[str, Any] | None = None,model_runtime: Mapping[str, Any] | None = None,model_config: Mapping[str, Any] | Any | None = None,requests_path: str | Path | None = None,results_path: str | Path | None = None,task_name: str | None = None,task_roots: Sequence[str | Path] | None = None,task_benchmark: str | None = None,task_recursive: bool = False,task_root_dir: str | Path | None = None,dataset_root: str | Path | None = None,dataset_id: str | None = None,split: str = 'default',num_samples: int | None = None,generated_artifact_dir: str | Path | None = None,output_artifact: str | None = None,required_artifacts: Sequence[str] | None = None,metrics: Sequence[str] = ('artifact_count', 'required_artifacts_present'),generation_cache_dir: str | Path | None = None,generation_cache_mode: str = 'off',generation_cache_namespace: str = 'worldfoundry_run',benchmark_timeout_seconds: float | None = None,benchmark_workdir: str | Path | None = None,benchmark_env: Mapping[str, Any] | None = None,benchmark_parameters: Mapping[str, Any] | None = None,materialize_placeholders: bool | None = None,contract_fixture: bool = False,fail_on_generation_error: bool = False,run_id: str | None = None,fail_on_sample_error: bool = False,write_artifacts_index: bool = True)

worldfoundry.evaluation.publicfrom worldfoundry.evaluation.public import WorldFoundryRunRequest

在进程内启动 WorldFoundry 评测的顶层请求（模型、任务、限制与输出目录）。

属性

output_dirstr | Path

model_idsSequence[str]默认值 ()

benchmark_idsSequence[str]默认值 ()

suite_idsSequence[str]默认值 ()

all_benchmarksbool默认值 False

benchmark_idstr | None默认值 None

benchmark_manifest_dirstr | Path默认值 BENCHMARK_ZOO_DIR

model_manifest_dirstr | Path | None默认值 MODEL_ZOO_DIR

suite_preset_pathstr | Path | None默认值 None

enginestr默认值 'auto'

benchmark_modestr默认值 'official-run'

executebool默认值 True

model_workersint默认值 1

worker_cuda_devicesSequence[str]默认值 ()

resumebool默认值 False

skip_incompatiblebool默认值 True

fail_on_skippedbool默认值 False

model_runnerstr | None默认值 None

model_variant_idstr | None默认值 None

model_parametersMapping[str, Any] | None默认值 None

model_runtimeMapping[str, Any] | None默认值 None

model_configMapping[str, Any] | Any | None默认值 None

requests_pathstr | Path | None默认值 None

results_pathstr | Path | None默认值 None

task_namestr | None默认值 None

task_rootsSequence[str | Path] | None默认值 None

task_benchmarkstr | None默认值 None

task_recursivebool默认值 False

task_root_dirstr | Path | None默认值 None

dataset_rootstr | Path | None默认值 None

dataset_idstr | None默认值 None

splitstr默认值 'default'

num_samplesint | None默认值 None

generated_artifact_dirstr | Path | None默认值 None

output_artifactstr | None默认值 None

required_artifactsSequence[str] | None默认值 None

metricsSequence[str]默认值 ('artifact_count', 'required_artifacts_present')

generation_cache_dirstr | Path | None默认值 None

generation_cache_modestr默认值 'off'

generation_cache_namespacestr默认值 'worldfoundry_run'

benchmark_timeout_secondsfloat | None默认值 None

benchmark_workdirstr | Path | None默认值 None

benchmark_envMapping[str, Any] | None默认值 None

benchmark_parametersMapping[str, Any] | None默认值 None

materialize_placeholdersbool | None默认值 None

contract_fixturebool默认值 False

fail_on_generation_errorbool默认值 False

run_idstr | None默认值 None

fail_on_sample_errorbool默认值 False

write_artifacts_indexbool默认值 True

`WorldFoundryRunResult`

Wrapper 提供共同的 status、exit code、输出目录与 mode-specific delegate。to_dict() 会把常用 manifest 和 scorecard 路径提升到顶层，因此自动化代码不必为每种 run kind 单独分支。

类

`WorldFoundryRunResult`

源码

class WorldFoundryRunResult(schema_version: str,kind: str,status: str,exit_code: int,output_dir: Path,delegate: Any)

worldfoundry.evaluation.publicfrom worldfoundry.evaluation.public import WorldFoundryRunResult

run_worldfoundry 的结构化结果：manifest、scorecard 与逐样本 ledger 路径。

属性

schema_versionstr

kindstr

statusstr

exit_codeint

output_dirPath

delegateAny

方法

propok -> bool源码

ok — Return True if the run completed successfully with exit code 0.

参数

self

返回值

bool

methto_dict() -> dict[str, Any]源码

to_dict — Convert the run result and its delegate details into a serializable dictionary.

返回值

dict[str, Any]

`run_worldfoundry`

可以传 typed request、mapping 或 keyword arguments。推荐 typed request，因为编辑器能补全字段，拼错字段也能在执行前暴露。

函数

`run_worldfoundry`

源码

def run_worldfoundry(request: WorldFoundryRunRequest | Mapping[str, Any] | None = None,**kwargs: Any) -> WorldFoundryRunResult

worldfoundry.evaluation.publicfrom worldfoundry.evaluation.public import run_worldfoundry

规范的进程内入口：执行 WorldFoundryRunRequest 并返回证据产物。

参数

requestWorldFoundryRunRequest | Mapping[str, Any] | None默认值 None

kwargsAny

返回值

WorldFoundryRunResult

`list_video_benchmarks`

这个发现 helper 返回仓库 video benchmark catalog 中的 ID。返回列表并不表示所有 benchmark 都已在本机可运行；readiness 与资产需要单独检查。

函数

`list_video_benchmarks`

源码

def list_video_benchmarks(catalog_dir: str | Path | None = None) -> list[str]

worldfoundry.evaluation.publicfrom worldfoundry.evaluation.public import list_video_benchmarks

列出公开评测 facade 可用的已注册视频 benchmark。

参数

catalog_dirstr | Path | None默认值 None

返回值

list[str]

`run_benchmark`

Artifact 已经物化、需要进入 benchmark 专属路径时使用这个 facade。official-run 调用配置的官方 runtime，official-validation 执行 bounded validation，normalizer 导入调用者提供的 official-shaped result。

result = run_benchmark(
    "vbench",
    output_dir="tmp/vbench_run",
    generated_artifact_dir="runs/generated_videos",
    mode="official-run",
)

上面是实际 API 形态，但运行需要满足当前 VBench manifest 报告的资产、依赖、prompt 覆盖率与环境。Runner 可用不代表 leaderboard ready。

函数

`run_benchmark`

源码

def run_benchmark(benchmark_id: str,output_dir: str | Path,mode: str = 'official-run',generated_artifact_dir: str | Path | None = None,manifest_path: str | Path = BENCHMARK_ZOO_DIR,**kwargs: Any) -> Any

worldfoundry.evaluation.publicfrom worldfoundry.evaluation.public import run_benchmark

简介

通过公开 facade 运行指定 benchmark，输入输出均已规范化。

源码 docstring

Run a benchmark through the unified official runner stack.

Modes mirror `worldfoundry-eval zoo benchmark-run`:

`normalizer`: normalize caller-provided official results
`official-validation`: run the benchmark's bounded validation command
`official-run`: invoke upstream official runtime when assets are available

参数

benchmark_idstr

output_dirstr | Path

modestr默认值 'official-run'

generated_artifact_dirstr | Path | None默认值 None

manifest_pathstr | Path默认值 BENCHMARK_ZOO_DIR

kwargsAny

返回值

Any

`normalize_upstream_results`

上游 evaluator 已经生成结果文件时使用这个函数。它会围绕该文件创建 WorldFoundry 证据，但不会反过来证明 WorldFoundry 曾执行官方 evaluator。

函数

`normalize_upstream_results`

源码

def normalize_upstream_results(benchmark_id: str,results_path: str | Path,output_dir: str | Path,generated_artifact_dir: str | Path | None = None,manifest_path: str | Path = BENCHMARK_ZOO_DIR,**kwargs: Any) -> Any

worldfoundry.evaluation.publicfrom worldfoundry.evaluation.public import normalize_upstream_results

把上游/厂商结果载荷适配为 WorldFoundry 的 GenerationResult 记录。

参数

benchmark_idstr

results_pathstr | Path

output_dirstr | Path

generated_artifact_dirstr | Path | None默认值 None

manifest_pathstr | Path默认值 BENCHMARK_ZOO_DIR

kwargsAny

返回值

Any

`benchmark_integration_spec`

当 in-tree integration specification 存在时，这个 lookup 会返回它。Catalog 条目可以先于 integration spec 存在，所以返回 None 是正常的发现结果。

函数

`benchmark_integration_spec`

源码

def benchmark_integration_spec(benchmark_id: str) -> BenchmarkIntegrationSpec | None

worldfoundry.evaluation.publicfrom worldfoundry.evaluation.public import benchmark_integration_spec

描述上游 benchmark 的接入方式（入口、产物与环境需求）。

参数

benchmark_idstr

返回值

BenchmarkIntegrationSpec | None

调用 official runtime 前，请先在 Benchmark Hub查看 protocol 专属输入与 blocker。