Contracts and artifacts

Serializable request, result, and artifact types used across model execution and evaluation.

Last updated: Oct 7, 2026

On this page

ArtifactReflocal_path_for_urienrich_artifact_refGenerationRequestGenerationResultnormalize_generation_statusis_generation_result_successfulSerialization round trip

These contracts are stdlib-only dataclasses. They can cross process and environment boundaries as JSON, which is why evaluation can consume outputs without importing the model runtime that produced them.

Import the symbols on this page from worldfoundry.evaluation.api.

`ArtifactRef`

ArtifactRef points to an output without embedding its bytes. kind describes the semantic role, while sha256, size_bytes, MIME information, and media metadata make a local artifact inspectable and auditable.

artifact = ArtifactRef.from_uri(
    "runs/matrix-game-2/generated.mp4",
    kind="generated_video",
    media_metadata={"fps": 12, "width": 640, "height": 352},
)

class

`ArtifactRef`

Source

class ArtifactRef(uri: str,kind: str,sha256: str | None = None,size_bytes: int | None = None,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = ARTIFACT_REF_SCHEMA_VERSION)

worldfoundry.evaluation.apifrom worldfoundry.evaluation.api import ArtifactRef

A portable pointer to an artifact (path or URI) with optional size, hash, and media metadata. Use it to pass outputs across process boundaries without embedding file bytes.

Attributes

uristr

kindstr

sha256str | Nonedefault None

size_bytesint | Nonedefault None

mime_typestr | Nonedefault None

media_metadataMapping[str, Any]default <dict factory>

metadataMapping[str, Any]default <dict factory>

schema_versionstrdefault ARTIFACT_REF_SCHEMA_VERSION

Methods

cmeth

from_path(path: str | Path,kind: str,uri: str | None = None,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None,compute_hash: bool = True) -> 'ArtifactRef'

Source

Overview

Build a reference to a local file, hashing it unless `compute_hash=False`.

Source docstring

Build a reference to a local file, hashing it unless `compute_hash=False`.

Set `compute_hash=False to skip the full-file SHA-256 read for large artifacts (e.g. multi-GB videos); the cheap size_bytes` stat is always recorded.

Parameters

pathstr | Path

kindstr

uristr | Nonedefault None

mime_typestr | Nonedefault None

media_metadataMapping[str, Any] | Nonedefault None

metadataMapping[str, Any] | Nonedefault None

compute_hashbooldefault True

Returns

'ArtifactRef'

cmeth

from_bytes(data: bytes,uri: str,kind: str,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None) -> 'ArtifactRef'

Source

Public classmethod on this type.

Parameters

databytes

uristr

kindstr

mime_typestr | Nonedefault None

media_metadataMapping[str, Any] | Nonedefault None

metadataMapping[str, Any] | Nonedefault None

Returns

'ArtifactRef'

cmeth

from_uri(uri: str | Path,kind: str,base_dir: str | Path | None = None,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None,compute_hash: bool = True) -> 'ArtifactRef'

Source

Public classmethod on this type.

Parameters

uristr | Path

kindstr

base_dirstr | Path | Nonedefault None

mime_typestr | Nonedefault None

media_metadataMapping[str, Any] | Nonedefault None

metadataMapping[str, Any] | Nonedefault None

compute_hashbooldefault True

Returns

'ArtifactRef'

cmethfrom_dict(data: Mapping[str, Any]) -> 'ArtifactRef'Source

Public classmethod on this type.

Parameters

dataMapping[str, Any]

Returns

'ArtifactRef'

`local_path_for_uri`

Use this helper before treating a URI as a local file. HTTP, object-store, Hugging Face, and in-memory URIs return None; relative local paths can be resolved against base_dir.

function

`local_path_for_uri`

Source

def local_path_for_uri(uri: str | Path,base_dir: str | Path | None = None) -> Path | None

worldfoundry.evaluation.apifrom worldfoundry.evaluation.api import local_path_for_uri

Resolve a URI to a local filesystem path when possible. Returns None for remote or non-file schemes so callers do not treat HTTP/object-store URIs as local files.

Parameters

uristr | Path

base_dirstr | Path | Nonedefault None

Returns

Path | None

`enrich_artifact_ref`

This function adds hash and size evidence when an artifact reference resolves to an existing local file. Remote or missing artifacts remain unchanged.

function

`enrich_artifact_ref`

Source

def enrich_artifact_ref(artifact: ArtifactRef,base_dir: str | Path | None = None,compute_hash: bool = True) -> ArtifactRef

worldfoundry.evaluation.apifrom worldfoundry.evaluation.api import enrich_artifact_ref

Overview

Fill size and SHA-256 on an ArtifactRef when the URI resolves to an existing local file. Missing or remote refs are left unchanged.

Source docstring

Return an ArtifactRef with file hash/size when its URI points to a local file.

Set `compute_hash=False to only fill the cheap size_bytes` stat and skip the full-file SHA-256 read (useful for very large artifacts).

Parameters

artifactArtifactRef

base_dirstr | Path | Nonedefault None

compute_hashbooldefault True

Returns

ArtifactRef

`GenerationRequest`

A request represents one sample, not an entire batch. Put source media or text in inputs, actions or camera paths in controls, native sampling settings such as seed and FPS in generation_kwargs, and expected artifact shape in output_schema.

For example, two models can receive the same sample_id and action sequence even if their operators translate those controls into different native argument names.

class

`GenerationRequest`

Source

class GenerationRequest(sample_id: str,task_name: str | None = None,task_id: str | None = None,split: str = 'default',request_id: str | None = None,inputs: Mapping[str, Any] | None = None,controls: Mapping[str, Any] | None = None,generation_kwargs: Mapping[str, Any] | None = None,output_schema: Mapping[str, Any] | None = None,cache_policy: Mapping[str, Any] | None = None,schema_version: str = GENERATION_REQUEST_SCHEMA_VERSION)

worldfoundry.evaluation.apifrom worldfoundry.evaluation.api import GenerationRequest

One evaluation sample’s generation request: inputs, controls, sampling kwargs, and expected outputs. Prefer one request per sample rather than packing a whole batch.

Attributes

sample_idstr

task_namestr

splitstrdefault 'default'

request_idstr | Nonedefault None

inputsMapping[str, Any]default <dict factory>

controlsMapping[str, Any]default <dict factory>

generation_kwargsMapping[str, Any]default <dict factory>

output_schemaMapping[str, Any]default <dict factory>

cache_policyMapping[str, Any]default <dict factory>

schema_versionstrdefault GENERATION_REQUEST_SCHEMA_VERSION

Methods

proptask_id -> strSource

Public property on this type.

Parameters

self

Returns

str

cmethfrom_dict(data: Mapping[str, Any]) -> 'GenerationRequest'Source

Public classmethod on this type.

Parameters

dataMapping[str, Any]

Returns

'GenerationRequest'

`GenerationResult`

A result records what happened for the corresponding sample. Successful output belongs in artifacts; execution details belong in timings and metadata; a failure keeps its status and error instead of creating a placeholder artifact.

result = GenerationResult(
    sample_id=request.sample_id,
    model_id="matrix-game-2",
    artifacts={"generated_video": artifact},
    timings={"generation_seconds": 49.75},
)

class

`GenerationResult`

Source

class GenerationResult(sample_id: str,request_id: str | None = None,model_id: str = '',artifacts: Mapping[str, ArtifactRef] = <dict factory>,status: str = 'succeeded',error: str | None = None,timings: Mapping[str, Any] = <dict factory>,metadata: Mapping[str, Any] = <dict factory>,schema_version: str = GENERATION_RESULT_SCHEMA_VERSION)

worldfoundry.evaluation.apifrom worldfoundry.evaluation.api import GenerationResult

Normalized generation outcome for one sample: artifacts on success, or status/error on failure. Benchmarks and metrics consume this shape, not the model runtime.

Attributes

sample_idstr

request_idstr | Nonedefault None

model_idstrdefault ''

artifactsMapping[str, ArtifactRef]default <dict factory>

statusstrdefault 'succeeded'

errorstr | Nonedefault None

timingsMapping[str, Any]default <dict factory>

metadataMapping[str, Any]default <dict factory>

schema_versionstrdefault GENERATION_RESULT_SCHEMA_VERSION

Methods

cmethfrom_dict(data: Mapping[str, Any]) -> 'GenerationResult'Source

Public classmethod on this type.

Parameters

dataMapping[str, Any]

Returns

'GenerationResult'

`normalize_generation_status`

Runners may use nearby status words such as success, completed, or done. This helper normalizes text before the success predicate is applied; it does not inspect whether a referenced file exists.

function

`normalize_generation_status`

Source

def normalize_generation_status(status: Any) -> str

worldfoundry.evaluation.apifrom worldfoundry.evaluation.api import normalize_generation_status

Overview

Map nearby status strings (success/completed/done, …) onto a canonical status before success checks. Does not verify that artifact files exist.

Source docstring

Return the canonical lowercase generation status text.

Missing/blank statuses normalize to `"unknown"` (fail-closed): an incomplete result row written by an external runner must not be treated as a scoreable success. Callers that own an in-process success signal (e.g. a pipeline returning normally) should set the status explicitly.

Parameters

statusAny

Raw status value from a generation runner or result row.

Returns

str

`is_generation_result_successful`

A result is considered successful when its normalized status is in the accepted success set and error is empty. Artifact existence and benchmark eligibility are separate checks.

function

`is_generation_result_successful`

Source

def is_generation_result_successful(result: 'GenerationResult') -> bool

worldfoundry.evaluation.apifrom worldfoundry.evaluation.api import is_generation_result_successful

True when the normalized status is an accepted success value and error is empty. Artifact presence and leaderboard eligibility are separate checks.

Parameters

result'GenerationResult'

Generation result emitted by an runner or loaded from disk.

Returns

bool

Serialization round trip

All three contract classes inherit the shared JSON contract methods. Use to_json() for a ledger row and from_json() or from_dict() when loading it again.

payload = result.to_json()
restored = GenerationResult.from_json(payload)

assert restored.sample_id == result.sample_id
assert restored.artifacts["generated_video"].kind == "generated_video"

Serialization proves that the record is well formed. It does not prove that the model ran, the artifact is visually correct, or the result is leaderboard eligible; those stronger claims come from runtime validation and scorecard evidence.