Registry & reference

MetricRegistry

The default registry loads all built-in entries from BUILTIN_METRIC_REGISTRY_ENTRIES in registry.py.

Discovery

worldfoundry-eval metric list
worldfoundry-eval metric list --json

worldfoundry-eval metric show fid
worldfoundry-eval metric show clip-score --json

worldfoundry-eval metric validate fid clip_score has_artifact:generated_artifact

Registry keys are normalized case-insensitively with hyphens instead of underscores (clip_score, clip-score, and clipscore resolve to the same entry). Parameterized ids use a prefix:

• has_artifact:<name> — Example: has_artifact:generated_artifact. Returns 1 when the named artifact exists on a result.
• numeric:<name> — Example: numeric:reward. Emits the numeric field <name> from result metadata/scores.

Python API

from worldfoundry.evaluation.tasks.metrics.registry import (
    default_metric_registry,
    list_metric_registry_entries,
    validate_metric_ids,
    create_existing_results_metric,
)

# List all registered entries
for entry in list_metric_registry_entries():
    print(entry.id, entry.family, entry.aliases)

# Resolve alias → canonical id
registry = default_metric_registry()
entry = registry.get("clip-score")
canonical = registry.canonical_metric_id("numeric:reward")

# Bulk validation
payload = validate_metric_ids(["fid", "unknown_metric"])
assert payload["ok"] is False

# Build offline evaluator for scorecard runs
metric = create_existing_results_metric(
    metrics=["artifact_count", "has_artifact:video"],
    required_artifacts=["generated_artifact"],
)

Registered metrics

Direction: ↑ higher is better, ↓ lower is better.

Image distribution (family=distribution)

Set-level metrics comparing reference and generated image collections. Torch-fidelity metrics (fid/, kid/, inception_score/, …) share vendored code under _shared/vendor/torch_fidelity/; standalone helpers include Clean-FID, CMMD, IPR, Vendi, Rarity, and fwd/vendor/pytorchfwd.

from worldfoundry.evaluation.tasks.metrics import (
    compute_distribution_metrics,
    compute_fid,
    compute_inception_score,
    compute_clip_fid,
    compute_cmmd,
    compute_clean_fid,
    compute_fwd,
    compute_improved_precision_recall,
    compute_vendi_score,
    compute_mean_rarity_score,
)

# Batch via torch-fidelity flags: isc, fid, kid, prc
scores = compute_distribution_metrics(
    "/path/to/reference",
    "/path/to/generated",
    metrics=("fid", "kid", "prc"),
)

fid = compute_fid("/path/to/reference", "/path/to/generated")
clip_fid = compute_fid("/path/to/reference", "/path/to/generated", feature_extractor="clip-vit-b-32")
swav_fid = compute_fid("/path/to/reference", "/path/to/generated", feature_extractor="swav-resnet50")
scene_fid = compute_scene_fid("/path/ref", "/path/gen", reference_bboxes_json="...")
is_scores = compute_inception_score("/path/to/generated")
cmmd = compute_cmmd("/path/to/reference", "/path/to/generated")
clean_fid = compute_clean_fid("/path/to/reference", "/path/to/generated")
fwd = compute_fwd("/path/to/reference", "/path/to/generated")
ipr = compute_improved_precision_recall("/path/to/reference", "/path/to/generated")
vendi = compute_vendi_score(feature_matrix)

FID, KID, Inception Score, precision/recall, MIND, and compute_distribution_metrics(...) accept image directories or explicit image-file lists. File lists preserve caller order and use the same RGB conversion and samples_resize_and_crop preprocessing as directories. SwAV FID also accepts file lists and applies its existing preprocessing and max_size limit.

The bundled backend loads under worldfoundry.evaluation.tasks.metrics._shared.vendor.torch_fidelity, independently of an installed torch_fidelity package or TorchMetrics import order. Register custom extractors and noise sources on that bundled package. PPL also accepts generator wrappers from the installed package, preserving their latent and class metadata.

compute_realism_score(...) accepts a generated image directory, an image file, an array, or a sequence of files and arrays. Directory batches produce one realism score per image; mean_realism is the mean of those scores and does not depend on batch size.

Precision measures the fraction of generated samples covered by the reference manifold; recall measures the fraction of reference samples covered by the generated manifold. In compute_precision_recall(...) and compute_distribution_metrics(...), backend input1_* options refer to generated inputs and input2_* options to reference inputs. The combined helper also computes Inception Score on the generated set.

compute_ppl(...) returns the backend's mean and standard deviation. With ppl_reduction="none", it also preserves the NumPy array perceptual_path_length_raw of per-sample distances after percentile filtering. summarize_distribution_metrics(...) renames this key to ppl_raw while preserving the array.

PPL defaults to the 1st–99th percentile range. Set ppl_discard_percentile_lower or ppl_discard_percentile_higher to None to disable that bound independently. For generator instances and module wrappers, conditional sampling uses the model's num_classes; paired latent samples receive the same label.

SceneFID manifest keys are image paths relative to the image root, or absolute paths. When generated_bboxes_json is omitted, reference keys are reused relative to the generated root, preserving subdirectories; absolute reference keys must be under the reference root. Missing images raise an error instead of falling back to another file with the same basename. Supply a separate generated manifest when filenames or boxes differ.

SceneFID, LQS, and object-wise consistency share one bbox convention: four values mean (x1, y1, x2, y2) and five values mean (x, y, width, height, extra). The fifth value does not affect geometry. Pixel and normalized coordinates use the same format; convert four-value xywh inputs before calling these metrics.

LQS matches objects within each category by minimum total center distance, including when object counts differ. Recall is matched objects divided by ground-truth objects; precision is matched objects divided by predicted objects. Location and area consistency use the matched pairs. This follows the matching objective and object-count denominators in the LQS definition, using SciPy's rectangular assignment solver instead of enumerating permutations.

Video distribution (family=distribution, video tags)

import numpy as np
from worldfoundry.evaluation.tasks.metrics import (
    compute_fvd_from_numpy,
    compute_fvd_from_frame_dirs,
    compute_fid_vid,
    compute_fvmd,
    compute_jedi_from_features,
)

# uint8 arrays: (N, T, H, W, C)
fvd = compute_fvd_from_numpy(real_videos, gen_videos, device="cuda")

fvd = compute_fvd_from_frame_dirs(
    reference_frame_dirs=["/path/to/ref_frames/video_0"],
    generated_frame_dirs=["/path/to/gen_frames/video_0"],
)

fid_vid = compute_fid("/path/to/ref_frames", "/path/to/gen_frames")  # alias: fid_vid
fvmd = compute_fvmd("/path/to/reference_videos", "/path/to/generated_videos")

jedi = compute_jedi_from_features(train_features, test_features)

JEDiMetric.load_features(num_samples=...) applies the same per-side sample cap to cached arrays and newly extracted features. Reading a cache does not truncate the stored file. Feature extraction runs locally without creating or destroying a distributed process group, and releases its extractor after loading both sides.

JEDi video batches use (B, T, C, H, W) with pixels in [0, 1]. The extractor converts them to channel-first clips before normalization. Encoder and probe checkpoints must match the model; missing or incompatible weights raise an error instead of retaining initialized weights.

For precomputed features, run_jedi_scorer(...) reports the actual reference_num_samples, generated_num_samples, and feature_dim. The legacy num_samples field contains the common count when both sides have equal lengths, or null when they differ.

The mock JEDi backend converts UTF-8 seed bytes directly into a NumPy seed. Identical seeds and dimensions reproduce the same features and score across fresh processes.

Perceptual pairwise (family=perceptual)

Pairwise similarity between reference and generated images (condition consistency, reconstruction quality).

import numpy as np
from worldfoundry.evaluation.tasks.metrics import (
    compute_lpips,
    compute_ssim,
    compute_ms_ssim,
    compute_psnr,
    compute_perceptual_bundle,
    compute_dino_similarity,
    compute_dreamsim,
)

ref = np.load("reference.npy")  # HxWxC uint8 or float
gen = np.load("generated.npy")

bundle = compute_perceptual_bundle(ref, gen)  # lpips, ssim, ms_ssim, psnr
dino_sim = compute_dino_similarity(ref, gen)
dreamsim_dist = compute_dreamsim(ref, gen)

PSNR, SSIM, MS-SSIM, and FSIM preserve the input pixel scale; both images must use the same units. data_range defaults to 255 if either input is uint8 or either maximum exceeds 1, otherwise 1. Pass an explicit range when float pixel values alone cannot identify the intended range; it is forwarded in the original units.

LPIPS accepts uint8 pixels or normalized floats in [0, 1], using TorchMetrics with normalize=True. For compatibility, float pairs with a maximum above 1 are also treated as [0, 255]. Both images always share one normalization scale.

CLI surface

• worldfoundry-eval metric list — Print or JSON-export all registry entries.
• worldfoundry-eval metric show <id> — Show one entry, resolved aliases, parameterized prefix.
• worldfoundry-eval metric validate <id>... — Bulk-resolve ids; exit 1 on unknown ids.
• worldfoundry-eval evaluate --metric ... — Built-in existing-results metrics only (artifact_count, required_artifacts_present, has_artifact:<name>, numeric, numeric:<name>).
• worldfoundry-eval plan create --metric ... — Same built-in metric IDs when constructing a run plan JSON.

Distribution and perceptual metrics are invoked through Python today. Benchmark runners may call the same underlying implementations when computing official metric stacks (EvalCrafter IS/FID paths, MiraBench FVD, and similar). Multimodal scorers (CLIPScore, VQAScore, ITMScore) are exposed through worldfoundry.evaluation.tasks.metrics.get_score_model(...) and implemented in execution/runners/_scorers/.

Dependencies

Install the distribution-metrics extra for torch-fidelity, Clean-FID, CMMD, perceptual torchmetrics, and related compute paths:

pip install -e ".[distribution_metrics]"

Core dependencies in the distribution_metrics extra include torch, torchvision, torchmetrics, scikit-learn, scipy, transformers, pillow, and the optional metric packages declared in pyproject.toml.

Keep heavy imports lazy: importing worldfoundry.evaluation.tasks.metrics should not require every checkpoint to be present until a specific compute_* function is called.

When changing a metric, compare representative inputs and scores with the upstream implementation, then record the backend version and results alongside the change. The private metric regression suite is not included in the public repository.

Checkpoints and environment variables

FVD resolution order is documented in fvd/fvd_core.py: explicit i3d_checkpoint argument → WORLDFOUNDRY_FVD_I3D_CKPT → WORLDFOUNDRY_MIRABENCH_FVD_I3D_CKPT → paths under WORLDFOUNDRY_CKPT_DIR.

Architecture

worldfoundry/evaluation/tasks/metrics/
├── _base.py             # MetricModuleSpec protocol + registry-entry factory
├── _discover.py         # Auto-discover METRIC_MODULE exports from per-metric folders
├── _shared/             # Cross-metric infra (distribution helpers, torch-fidelity loader)
│   └── vendor/torch_fidelity/  # Vendored torch-fidelity (Apache-2.0)
├── registry.py          # MetricRegistryEntry, MetricRegistry, legacy + discovered entries
├── builtins.py          # BuiltinExistingResultsMetric (offline result scoring)
├── fid/ kid/ …          # One folder per distribution metric (METRIC_ID, compute, METRIC_MODULE)
├── lpips/ ssim/ …       # One folder per pairwise perceptual metric
├── fvd/ fvmd/ opens2v/ sadpad/  # Video distribution + OpenS2V + SadPaD attributes
├── cmmd/ clean_fid/ vendi_score/ …      # Standalone distribution helpers
└── jedi/                # Video JEPA distance (wrapper.py + vendored JEDi)

Per-metric module layout

Each migrated metric folder exports a unified surface:

• METRIC_ID, ALIASES, HIGHER_IS_BETTER, FAMILY, TAGS — Registry metadata
• METRIC_MODULE — :class:MetricModuleSpec for auto-discovery
• compute(...) / compute_<name>(...) — Stable compute API by metric family
• Paper / repo links — Listed inline under each metric in metric recipe pages

Registry entries for migrated metrics are registered via _discover.discover_metric_registry_entries() instead of hard-coded duplicates in registry.py.

Import surface

Top-level metrics/__init__.py re-exports compute_* functions from per-metric modules so existing from worldfoundry.evaluation.tasks.metrics import compute_fid imports keep working.

Benchmark catalog metrics and official-result normalization belong in execution/runners/; benchmark-specific setup belongs on the matching Benchmark Hub page.

Related pages

• Evaluation — run contract, evaluate, and scorecard outputs.
• Benchmark Hub — benchmark-specific metrics, assets, and run commands.
• Local asset preparation — checkpoint staging including FVD I3D weights.