WorldFoundry
English中文

Metrics

In-tree metric registry and distribution/perceptual compute APIs.

Edit this pageGitHubMarkdown

On this page
Getting startedPagesRelated pages

WorldFoundry ships an in-tree metric layer — a registry of scorers and distribution/perceptual compute routines that runners invoke through a stable contract. Benchmarks decide what to measure on which data; metrics decide how to turn predictions into numbers. Each entry declares inputs, optional checkpoint dependencies, and output shape; runners reference metric IDs and the registry resolves aliases, validates availability, and routes to the right compute_*. This keeps model code out of scoring while making scoring reproducible across machines. This hub documents what the repository actually implements — it is not a survey of every published score.

The in-tree catalog spans four families: scorers & quality (CLIP, VQA, ITM, face, art, S2V judges), distribution (FID, IS, KID, FVD, JEDi, CMMD, Clean-FID), perceptual pairwise (LPIPS, SSIM, DINO, mask, layout), and editing & layout (SemSR, IRS, CAS, manipulation). Metrics that need pretrained backbones declare those dependencies explicitly — missing checkpoints fail loudly, never silently substitute a proxy. A fifth page, Registry & reference, covers discovery (worldfoundry-eval metric list), parameterized ids, checkpoint staging, and package layout. For benchmark-specific evaluator assets and commands, see Benchmark Hub; individual pages call out approximations and cases where official scripts must be used instead of the generic compute API.

Getting started

Three steps:

  1. source WorldFoundry/tmp/worldfoundry_unified_env.sh and pip install -e ".[distribution_metrics]"
  2. worldfoundry-eval metric show <id> to resolve ids and aliases
  3. Open the metric page below and copy the recipe for your id

Only built-in existing-results metrics use worldfoundry-eval evaluate; everything else is Python (compute_* or scorer classes).

Metric imports are lazy: checkpoints are required only when a metric loads a feature extractor, multimodal scorer, or API judge. Default caches follow Hugging Face / Torch / torchvision layout:

export WORLDFOUNDRY_HFD_ROOT=${HF_HOME:-$HOME/.cache/huggingface}
export WORLDFOUNDRY_T2V_METRICS_CACHE_DIR=$WORLDFOUNDRY_HFD_ROOT
export WORLDFOUNDRY_CKPT_DIR=$HOME/.cache/worldfoundry/checkpoints

Pages

  • Scorers & quality — CLIPScore, VQAScore, ITMScore, FaceScore, ArtScore, OpenS2V reward metrics.
  • Distribution — FID, IS, KID, FVD, JEDi, CMMD, Clean-FID, and feature-array metrics.
  • Perceptual pairwise — LPIPS, SSIM, DINO similarity, mask accuracy, layout quality.
  • Editing & layout — SemSR, IRS, CAS, manipulation direction, object-wise consistency.
  • Registry & reference — MetricRegistry, registered-metric tables, CLI surface, checkpoints, architecture.