Runs and benchmarks

run_worldfoundry is the broad public entrypoint. It examines one typed request and dispatches to the narrow existing-results, model-generation, single model × benchmark, or suite runner. Use the narrower benchmark facade when the generated artifacts already exist and only an official evaluator or normalizer should run.

Import the symbols on this page from worldfoundry.evaluation.public.

A complete no-GPU run

This example creates one small trajectory artifact, writes request and result ledgers, and evaluates the existing result with the built-in artifact_count metric. It exercises the real run/reporting path without loading a model.

from pathlib import Path

from worldfoundry.evaluation.api import ArtifactRef, GenerationRequest, GenerationResult
from worldfoundry.evaluation.public import WorldFoundryRunRequest, run_worldfoundry

root = Path("tmp/python_api_example")
trace_path = root / "artifacts" / "trajectory.json"
trace_path.parent.mkdir(parents=True, exist_ok=True)
trace_path.write_text('{"actions":["forward","left"]}\n', encoding="utf-8")

request = GenerationRequest(sample_id="nav-0001", task_name="navigation-trace")
result = GenerationResult(
    sample_id=request.sample_id,
    model_id="existing-trace",
    artifacts={
        "trajectory": ArtifactRef.from_path(trace_path, kind="trajectory"),
    },
)

requests_path = root / "requests.jsonl"
results_path = root / "results.jsonl"
requests_path.write_text(request.to_json() + "\n", encoding="utf-8")
results_path.write_text(result.to_json() + "\n", encoding="utf-8")

outcome = run_worldfoundry(
    WorldFoundryRunRequest(
        output_dir=root / "evaluation",
        requests_path=requests_path,
        results_path=results_path,
        metrics=("artifact_count",),
    )
)

assert outcome.ok
print(outcome.to_dict()["scorecard_path"])

The output directory contains aligned ledgers, an execution plan, metric rows, run_manifest.json, summary.json, report.md, and scorecard.json. The run is valid as an existing-results artifact check; it does not become an official benchmark or leaderboard result.

WorldFoundryRunRequest

The request intentionally covers several modes. results_path selects existing-results evaluation. A model ID without benchmark IDs selects model execution. Model and benchmark IDs together select a benchmark cell; multiple selections or suite IDs select the matrix runner. execute=False plans compatible cells without spending compute.

class WorldFoundryRunRequest(output_dir: str | Path,model_ids: Sequence[str] = (),benchmark_ids: Sequence[str] = (),suite_ids: Sequence[str] = (),all_benchmarks: bool = False,benchmark_id: str | None = None,benchmark_manifest_dir: str | Path = BENCHMARK_ZOO_DIR,model_manifest_dir: str | Path | None = MODEL_ZOO_DIR,suite_preset_path: str | Path | None = None,engine: str = 'auto',benchmark_mode: str = 'official-run',execute: bool = True,model_workers: int = 1,worker_cuda_devices: Sequence[str] = (),resume: bool = False,skip_incompatible: bool = True,fail_on_skipped: bool = False,model_runner: str | None = None,model_variant_id: str | None = None,model_parameters: Mapping[str, Any] | None = None,model_runtime: Mapping[str, Any] | None = None,model_config: Mapping[str, Any] | Any | None = None,requests_path: str | Path | None = None,results_path: str | Path | None = None,task_name: str | None = None,task_roots: Sequence[str | Path] | None = None,task_benchmark: str | None = None,task_recursive: bool = False,task_root_dir: str | Path | None = None,dataset_root: str | Path | None = None,dataset_id: str | None = None,split: str = 'default',num_samples: int | None = None,generated_artifact_dir: str | Path | None = None,output_artifact: str | None = None,required_artifacts: Sequence[str] | None = None,metrics: Sequence[str] = ('artifact_count', 'required_artifacts_present'),generation_cache_dir: str | Path | None = None,generation_cache_mode: str = 'off',generation_cache_namespace: str = 'worldfoundry_run',benchmark_timeout_seconds: float | None = None,benchmark_workdir: str | Path | None = None,benchmark_env: Mapping[str, Any] | None = None,benchmark_parameters: Mapping[str, Any] | None = None,materialize_placeholders: bool | None = None,contract_fixture: bool = False,fail_on_generation_error: bool = False,run_id: str | None = None,fail_on_sample_error: bool = False,write_artifacts_index: bool = True)

WorldFoundryRunResult

The wrapper exposes common status, exit code, output directory, and a mode-specific delegate. to_dict() lifts commonly needed manifest and scorecard paths from that delegate so automation does not need a branch for every run kind.

class WorldFoundryRunResult(schema_version: str,kind: str,status: str,exit_code: int,output_dir: Path,delegate: Any)

run_worldfoundry

Pass either a typed request, a mapping, or keyword arguments. A typed request is preferred for editor support and for catching misspelled fields before execution.

def run_worldfoundry(request: WorldFoundryRunRequest | Mapping[str, Any] | None = None,**kwargs: Any) -> WorldFoundryRunResult

list_video_benchmarks

This discovery helper returns IDs from the checked-in video benchmark catalog. It does not claim that every returned benchmark is locally runnable; inspect readiness and assets separately.

def list_video_benchmarks(catalog_dir: str | Path | None = None) -> list[str]

run_benchmark

Use this facade when artifacts are already materialized and a benchmark-specific path should run. official-run invokes the configured official runtime, official-validation executes its bounded validation path, and normalizer imports caller-provided official-shaped results.

result = run_benchmark(
    "vbench",
    output_dir="tmp/vbench_run",
    generated_artifact_dir="runs/generated_videos",
    mode="official-run",
)

The call above is a real API shape, but it requires the assets, dependencies, generated prompt coverage, and environment reported by the current VBench manifest. Runner availability alone does not guarantee leaderboard readiness.

def run_benchmark(benchmark_id: str,output_dir: str | Path,mode: str = 'official-run',generated_artifact_dir: str | Path | None = None,manifest_path: str | Path = BENCHMARK_ZOO_DIR,**kwargs: Any) -> Any

normalize_upstream_results

Use this function when the upstream evaluator has already produced a result file. It creates WorldFoundry evidence around that file; it does not retroactively prove that WorldFoundry executed the official evaluator.

def normalize_upstream_results(benchmark_id: str,results_path: str | Path,output_dir: str | Path,generated_artifact_dir: str | Path | None = None,manifest_path: str | Path = BENCHMARK_ZOO_DIR,**kwargs: Any) -> Any

benchmark_integration_spec

This lookup returns the registered in-tree integration specification when one exists. A catalog entry can exist without a corresponding integration spec, so None is a normal discovery result.

def benchmark_integration_spec(benchmark_id: str) -> BenchmarkIntegrationSpec | None

Use the Benchmark Hub to understand protocol-specific inputs and blockers before calling an official runtime.