Metrics and tasks
Metric outputs, metric implementations, task contracts, evaluation protocols, and benchmark specifications.
On this page
Task contracts describe what should be generated. Metric contracts describe how compatible results are scored. Keeping them separate allows one generated artifact set to be evaluated by more than one metric without rerunning the model.
Import the symbols on this page from worldfoundry.evaluation.api.
MetricSpec
MetricSpec is declarative metadata: identity, aliases, required artifact kinds, output unit, aggregation policy, and score direction. The implementation field may point to code, but the spec itself does not execute it.
MetricSpec
class MetricSpec(id: str | None = None,metric_id: str | None = None,aliases: Sequence[str] = (),display_name: str = '',description: str = '',version: str = '1.0',family: str = '',capability: str = '',requires_reference: bool = False,required_artifacts: Sequence[str] = (),output_unit: str = '',higher_is_better: bool | None = None,normalizer: str | None = None,aggregator: str = 'mean',statistics: Sequence[str] = ('mean',),primary: bool = False,weight: float = 1.0,implementation: str | None = None,tags: Sequence[str] = (),metadata: Mapping[str, Any] | None = None,schema_version: str = METRIC_SPEC_SCHEMA_VERSION)Declarative description of a metric’s identity, inputs, and aggregation expectations used by registries and scorecards.
Attributes
idstraliasestuple[str, ...]default ()display_namestrdefault ''descriptionstrdefault ''versionstrdefault '1.0'familystrdefault ''capabilitystrdefault ''requires_referencebooldefault Falserequired_artifactstuple[str, ...]default ()output_unitstrdefault ''higher_is_betterbool | Nonedefault Nonenormalizerstr | Nonedefault Noneaggregatorstrdefault 'mean'statisticstuple[str, ...]default ('mean',)primarybooldefault Falseweightfloatdefault 1.0implementationstr | Nonedefault Nonetagstuple[str, ...]default ()metadataMapping[str, Any]default <dict factory>schema_versionstrdefault METRIC_SPEC_SCHEMA_VERSIONMethods
Public classmethod on this type.
Parameters
dataMapping[str, Any]Returns
MetricResult
One MetricResult belongs to one sample and one metric. Use valid=False with skip_reason when the metric cannot score the sample. coverage makes partial evaluation visible instead of silently averaging only the successful subset.
MetricResult
class MetricResult(sample_id: str,metric_id: str,raw_value: Any = None,normalized_value: float | None = None,components: Mapping[str, Any] = <dict factory>,valid: bool = True,coverage: float = 1.0,skip_reason: str | None = None,diagnostics: Mapping[str, Any] = <dict factory>,artifact_refs: Mapping[str, ArtifactRef] = <dict factory>,judge_trace: Mapping[str, Any] = <dict factory>,schema_version: str = METRIC_RESULT_SCHEMA_VERSION)Per-sample metric output with score payload and optional diagnostics. AggregateResult rolls many of these up.
Attributes
sample_idstrmetric_idstrraw_valueAnydefault Nonenormalized_valuefloat | Nonedefault NonecomponentsMapping[str, Any]default <dict factory>validbooldefault Truecoveragefloatdefault 1.0skip_reasonstr | Nonedefault NonediagnosticsMapping[str, Any]default <dict factory>judge_traceMapping[str, Any]default <dict factory>schema_versionstrdefault METRIC_RESULT_SCHEMA_VERSIONMethods
Public classmethod on this type.
Parameters
dataMapping[str, Any]Returns
AggregateResult
An aggregate records total, valid, and skipped counts together with summary statistics. It is the metric-level result; leaderboard eligibility is decided later by benchmark and scorecard evidence.
AggregateResult
class AggregateResult(metric_id: str,n_total: int = 0,n_valid: int = 0,n_skipped: int = 0,raw_stats: Mapping[str, Any] = <dict factory>,normalized_stats: Mapping[str, Any] = <dict factory>,confidence_interval: Mapping[str, Any] = <dict factory>,stderr: float | None = None,skip_breakdown: Mapping[str, int] = <dict factory>,valid: bool = True,diagnostics: Mapping[str, Any] = <dict factory>,schema_version: str = AGGREGATE_RESULT_SCHEMA_VERSION)Dataset- or split-level aggregation over MetricResult rows (means, counts, custom summaries).
Attributes
metric_idstrn_totalintdefault 0n_validintdefault 0n_skippedintdefault 0raw_statsMapping[str, Any]default <dict factory>normalized_statsMapping[str, Any]default <dict factory>confidence_intervalMapping[str, Any]default <dict factory>stderrfloat | Nonedefault Noneskip_breakdownMapping[str, int]default <dict factory>validbooldefault TruediagnosticsMapping[str, Any]default <dict factory>schema_versionstrdefault AGGREGATE_RESULT_SCHEMA_VERSIONMethods
Public classmethod on this type.
Parameters
dataMapping[str, Any]Returns
Metric
Metric implementations satisfy this protocol structurally. The following deliberately simple metric counts artifact references; it shows the API shape, not a benchmark-quality video metric.
from worldfoundry.evaluation.api import AggregateResult, MetricResult
class ArtifactCountMetric:
name = "artifact_count_example"
version = "1.0"
required_artifacts = ()
higher_is_better = None
def compute_sample(self, request, result):
value = len(result.artifacts)
return MetricResult(
sample_id=request.sample_id,
metric_id=self.name,
raw_value=value,
normalized_value=value,
)
def aggregate(self, results):
values = [item.raw_value for item in results if item.valid]
mean = sum(values) / len(values) if values else None
return AggregateResult(
metric_id=self.name,
n_total=len(results),
n_valid=len(values),
n_skipped=len(results) - len(values),
raw_stats={"mean": mean},
normalized_stats={"mean": mean},
valid=bool(values),
)Metric
class Metric(name: str,version: str,required_artifacts: tuple[str, ...],higher_is_better: bool | None)Minimum metric implementation surface: score samples and optionally aggregate. Keep heavyweight backends behind this contract.
Attributes
namestrversionstrrequired_artifactstuple[str, ...]higher_is_betterbool | NoneMethods
EvaluationProtocolSpec
A protocol groups metric IDs and metric groups under a named evaluation behavior. Extra protocol-specific keys loaded from a catalog are preserved in metadata.
EvaluationProtocolSpec
class EvaluationProtocolSpec(name: str,metric_ids: tuple[str, ...] = (),metric_groups: tuple[str, ...] = (),metadata: Mapping[str, Any] = <dict factory>,schema_version: str = EVALUATION_PROTOCOL_SCHEMA_VERSION)Overview
Describes how a task/protocol evaluates generations (required artifacts, metrics, pass rules).
Source docstring
Evaluation protocol entry for a task or benchmark catalog task.
The public API keeps simple string protocols supported, while catalog surfaces can use this structured form to attach metric ids, groups, and protocol-specific metadata.
Attributes
namestrmetric_idstuple[str, ...]default ()metric_groupstuple[str, ...]default ()metadataMapping[str, Any]default <dict factory>schema_versionstrdefault EVALUATION_PROTOCOL_SCHEMA_VERSIONMethods
Public classmethod on this type.
Parameters
dataMapping[str, Any]Returns
WorldTaskConfig
This object describes one task's input keys, output keys, generation defaults, and intended metric set. For an action-conditioned video task, inputs might contain an initial image, controls might carry actions, and the output key would be generated_video.
from worldfoundry.evaluation.api import WorldTaskConfig
task = WorldTaskConfig(
name="action-conditioned-video",
protocol="open_loop",
input_keys=("image", "actions"),
output_keys=("generated_video",),
metric_ids=("artifact_count_example",),
generation_defaults={"fps": 12, "seed": 42},
)WorldTaskConfig
class WorldTaskConfig(name: str | None = None,task_id: str | None = None,protocol: str = 'open_loop',evaluation_protocol: str = 'reference_metrics',capability_track: str = 'core_video',schema_type: str = 'sample',input_keys: Sequence[str] = (),output_keys: Sequence[str] = ('generated_video',),metric_ids: Sequence[str] = (),metric_groups: Sequence[str] = (),tags: Sequence[str] = (),description: str = '',data: Mapping[str, Any] | None = None,generation_defaults: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = WORLD_TASK_CONFIG_SCHEMA_VERSION)Task configuration bound to a protocol and dataset slice for one evaluation run.
Attributes
namestrprotocolstrdefault 'open_loop'evaluation_protocolstrdefault 'reference_metrics'capability_trackstrdefault 'core_video'schema_typestrdefault 'sample'input_keystuple[str, ...]default ()output_keystuple[str, ...]default ('generated_video',)metric_idstuple[str, ...]default ()metric_groupstuple[str, ...]default ()tagstuple[str, ...]default ()descriptionstrdefault ''dataMapping[str, Any]default <dict factory>generation_defaultsMapping[str, Any]default <dict factory>metadataMapping[str, Any]default <dict factory>schema_versionstrdefault WORLD_TASK_CONFIG_SCHEMA_VERSIONMethods
Public classmethod on this type.
Parameters
dataMapping[str, Any]Returns
BenchmarkSpec
BenchmarkSpec collects tasks, metrics, splits, and dataset metadata into an in-process benchmark description. A checked-in benchmark catalog entry can be richer; this public DTO is the execution-facing form.
from worldfoundry.evaluation.api import BenchmarkSpec, MetricSpec
benchmark = BenchmarkSpec(
name="navigation-smoke-test",
tasks=(task,),
metrics=(MetricSpec(id="artifact_count_example"),),
splits=("validation",),
)BenchmarkSpec
class BenchmarkSpec(name: str | None = None,benchmark_id: str | None = None,version: str = '1.0',tasks: Sequence[WorldTaskConfig | Mapping[str, Any]] = (),metrics: Sequence[MetricSpec | Mapping[str, Any]] = (),splits: Sequence[str] = ('default',),tags: Sequence[str] = (),description: str = '',dataset_root: str | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = BENCHMARK_SPEC_SCHEMA_VERSION)Public benchmark identity and wiring metadata used by the hub and runners.
Attributes
namestrversionstrdefault '1.0'splitstuple[str, ...]default ('default',)tagstuple[str, ...]default ()descriptionstrdefault ''dataset_rootstr | Nonedefault NonemetadataMapping[str, Any]default <dict factory>schema_versionstrdefault BENCHMARK_SPEC_SCHEMA_VERSIONMethods
Public classmethod on this type.
Parameters
dataMapping[str, Any]Returns
For an official benchmark integration, these contracts are necessary but not sufficient. The add a benchmark guide covers datasets, official runners, normalizers, runtime profiles, coverage, and evidence gates.