---
title: CLI reference
description: Complete command map, execution modes, output contracts, and automation conventions for worldfoundry-eval.
---

`worldfoundry-eval` is the canonical command-line entrypoint for WorldFoundry. It covers catalog discovery, readiness checks, typed model inference, benchmark execution, result normalization, reporting, and validation. Use the [Quickstart](/docs/quickstart) for the first clone-to-run walkthrough; use this page when choosing a command or checking its contract.

<Callout type="info" title="Safe to inspect">
  Help, catalog discovery, `--model-status`, `--print-config`, `--plan-only`, and local asset checks do not load model weights. They are the right place to start on a CPU-only machine or before allocating a GPU job.
</Callout>

## Install and verify

Run commands from the repository root after activating the unified environment.

<Tabs items={['Recommended', 'Editable install', 'Module fallback']}>
  <Tab>

```bash
bash scripts/setup/bootstrap_worldfoundry.sh
source tmp/worldfoundry_unified_env.sh
conda activate "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}"
worldfoundry-eval --help
```

`bootstrap_worldfoundry.sh` is the official first-clone script used in the [Quickstart](/docs/quickstart), TUI, and Studio pages.

  </Tab>
  <Tab>

For catalog and core CLI development in an existing Python environment:

```bash
python -m pip install -e .

# Add an optional surface only when needed.
python -m pip install -e ".[tui]"
python -m pip install -e ".[mcp]"
```

Model and benchmark runtimes can still require the environments and assets declared by their profiles.

  </Tab>
  <Tab>

If the console script is not on `PATH`, the module entrypoint uses the same parser:

```bash
python -m worldfoundry --help
python -m worldfoundry zoo models --json
```

  </Tab>
</Tabs>

Running `worldfoundry-eval` with no arguments prints a short first-run banner. Add `--help` for the complete top-level command list.

## Entrypoints

- **`worldfoundry-eval`** — Canonical CLI used throughout the documentation.
- **`worldfoundry`** — Alias of `worldfoundry-eval`; it dispatches to the same `main()` function.
- **`python -m worldfoundry`** — Module fallback when an editable install or console-script `PATH` is unavailable.
- **`worldfoundry-tui`** — Dedicated alias for the terminal UI; equivalent to `worldfoundry-eval tui`.
- **`worldfoundry-mcp`** — Dedicated MCP server entrypoint; `worldfoundry-eval mcp` reaches the same surface.
- **`worldfoundry-studio`** — Separate Studio launcher. See the [Studio guide](/docs/guides/studio).
- **`python -m worldfoundry.studio.workspace_job infer`** — Workspace visual smoke (the same job path the Studio UI submits). Not the documented first-run command.

**Which command.** Everyday reproducible / model-page commands use `worldfoundry-eval run <id> --pipeline.task-profile …`. Use `python -m worldfoundry.studio.workspace_job infer` only for Workspace visual smoke. Run-status labels are in [Run status](/docs/guides/run-status).

Examples below use `worldfoundry-eval` so shell history, job scripts, and bug reports share one spelling.

## Help at every level

Append `--help` after the command whose flags you need. Options belong to the leaf command, so `--json` is written after `zoo models`, not before it.

```bash
worldfoundry-eval --help
worldfoundry-eval zoo --help
worldfoundry-eval zoo model-show --help
worldfoundry-eval run --help
```

![Terminal output of worldfoundry-eval --help](/images/cli/worldfoundry-eval-help.png)

<div className="text-sm text-fd-muted-foreground">

Captured from the current CLI. The screenshot can be regenerated with `npm run cli:screenshots` in `docs/fumadocs`.

</div>

### Model-specific help

The positional form `run MODEL` loads that model's inference schema and adds typed `--pipeline.*`, `--pipeline.load.*`, and `--runtime.*` options to its help page.

```bash
worldfoundry-eval run self-forcing --help
worldfoundry-eval run self-forcing --model-status
worldfoundry-eval run self-forcing --print-config --json
```

- `--pipeline.*` controls request inputs and generation defaults.
- `--pipeline.load.*` controls checkpoint and model-loading fields.
- `--runtime.*` controls the execution device and runner settings.
- `--model-status` and `--print-config` resolve metadata only; they do not load weights.

![Terminal output showing one model-zoo readiness record](/images/cli/model-readiness.png)

## Choose the right command

Several commands can reach the evaluation core, but they start from different user inputs.

- **Browse stable model or benchmark IDs** — prefer `zoo models`, `zoo benchmarks`, `zoo model-show`, `zoo benchmark-show`. Reads the release catalogs and surfaces readiness, aliases, needs, and next actions.
- **Run one model directly** — prefer `run MODEL`. Adds model-specific typed inference flags.
- **Run one or many model × benchmark cells** — prefer `run --model ... --benchmark ...`. Unified facade for a cell, repeated IDs, named suites, resume, cache, and plan-only execution.
- **Score an artifact directory with a benchmark** — prefer `score --benchmark ... --artifacts ...`. User-oriented scoring intent with a complete benchmark protocol.
- **Score WorldFoundry result rows with explicit metrics** — prefer `score --results ... --metric ...`. Compact existing-results workflow.
- **Execute a materialized run plan or use eval-core fields directly** — prefer `evaluate` (alias `eval`). Lower-level deterministic path for requests/results ledgers and model-mode evaluation.
- **Generate from a dataset manifest, then score** — prefer `generate-score`. One intent owns dataset materialization, generation, cache, and metrics.
- **Import or execute one benchmark's official surface** — prefer `zoo benchmark-run`. Exposes benchmark-specific modes and parameters. Follow that benchmark's Hub page.
- **Re-run a pinned model × benchmark recipe** — prefer `reproduce`. Resolves a recipe YAML.
- **Build indexes, comparisons, or contract checks** — prefer `index-runs`, `compare-runs`, `validate-artifact`. Operates on durable run outputs without rerunning a model.

## Complete command map

<div className="pi-kv-catalog not-prose">
  <section className="pi-kv-group">
    <h3>Discovery and interactive surfaces</h3>
    <ul>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>tui</code>
          </div>
        </div>
        <p>Interactive catalog browser and command builder.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>zoo</code>
          </div>
        </div>
        <p><code>models</code>, <code>model-specs</code>, <code>model-show</code>, <code>model-download</code>, <code>embodied-assets</code>, <code>benchmarks</code>, <code>benchmark-specs</code>, <code>benchmark-show</code>, <code>benchmark-run</code>.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>tasks</code>
          </div>
        </div>
        <p><code>list</code>, <code>show</code>, <code>catalog</code> for registered benchmark tasks.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>suites</code>
          </div>
        </div>
        <p><code>list</code>, <code>show</code> for named model × benchmark presets.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>models</code>
          </div>
        </div>
        <p><code>list</code>, <code>runtime-runners</code>, <code>visualizations</code>, <code>assets</code>, <code>visualize</code>.</p>
      </li>
    </ul>
  </section>
  <section className="pi-kv-group">
    <h3>Execution and scoring</h3>
    <ul>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>run</code>
          </div>
        </div>
        <p>Direct typed inference, one model × benchmark cell, or a suite/matrix.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>score</code>
          </div>
        </div>
        <p>Score an artifact directory with a benchmark, or result rows with selected metrics.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>generate-score</code>
          </div>
        </div>
        <p>Run an in-tree model over a dataset manifest and score the results.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>reproduce</code>
          </div>
        </div>
        <p>Execute a recipe YAML.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>evaluate</code>
            <code>eval</code>
          </div>
        </div>
        <p>Deterministic eval-core execution for materialized results, run plans, or model mode.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>embodied</code>
          </div>
        </div>
        <p><code>plan</code>, <code>run</code>, <code>serve</code>, <code>merge</code> for simulator-backed closed-loop evaluation.</p>
      </li>
    </ul>
  </section>
  <section className="pi-kv-group">
    <h3>Data, plans, and validation</h3>
    <ul>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>task</code>
          </div>
        </div>
        <p><code>list</code>, <code>show</code>, <code>validate</code>, <code>materialize</code> filesystem task YAML.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>dataset</code>
          </div>
        </div>
        <p><code>create</code>, <code>show</code>, <code>validate</code>, <code>materialize</code> sample manifests; <code>locate</code>, <code>plan</code> local/HF dataset assets.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>config</code>
          </div>
        </div>
        <p><code>list</code>, <code>run</code> checked-in workflow templates.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>plan</code>
          </div>
        </div>
        <p><code>create</code>, <code>show</code>, <code>validate</code> stable <code>worldfoundry-run-plan</code> JSON.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>metric</code>
          </div>
        </div>
        <p><code>list</code>, <code>show</code>, <code>validate</code> executable metric IDs.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>preflight runtime</code>
          </div>
        </div>
        <p>Check imports, environment variables, paths, CUDA, and declared validation gaps without running a benchmark.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>validate</code>
          </div>
        </div>
        <p>Validate legacy benchmark metadata loading with a concrete task/data selector.</p>
      </li>
    </ul>
  </section>
  <section className="pi-kv-group">
    <h3>Reports and integrations</h3>
    <ul>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>index-runs</code>
          </div>
        </div>
        <p>Build <code>index.json</code>, <code>index.jsonl</code>, and an optional dependency-free HTML browser.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>compare-runs</code>
          </div>
        </div>
        <p>Compare explicit run directories or select runs from an index.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>validate-artifact</code>
          </div>
        </div>
        <p>Validate summary, scorecard, index, comparison, and suite schemas.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>mcp</code>
          </div>
        </div>
        <p>Start the MCP server (<code>--transport stdio|sse</code>, default <code>stdio</code>).</p>
      </li>
    </ul>
  </section>
</div>

### MCP tools

`worldfoundry-eval mcp` and `worldfoundry-mcp` start the same FastMCP server (`worldfoundry/mcp/server.py`). After `python -m pip install -e ".[mcp]"`:

```bash
worldfoundry-eval mcp --transport stdio
worldfoundry-eval mcp --transport sse
```

The server registers 27 tools: `server_info`; discovery (`list_models`, `get_model_info`, `list_benchmarks`, `get_benchmark_info`, `list_tasks`, `get_task_info`, `list_metrics`, `show_metric`, `check_benchmark_datasets`); run lifecycle (`preview_run`, `evaluate`, `list_runs`, `get_run_status`, `get_run_result`, `get_run_samples`, `cancel_run`); and Studio workspace (`list_studio_models`, `get_studio_model_info`, `submit_studio_inference`, `list_studio_jobs`, `get_studio_job`, `wait_for_studio_job`, `stop_studio_job`, `get_studio_job_logs`, `list_studio_artifacts`, `get_studio_manifest`). Studio tools default to `WORLDFOUNDRY_STUDIO_WORKSPACE_URL` (typically `http://127.0.0.1:7870`). Run files land under `WORLDFOUNDRY_MCP_RUN_ROOT`, or `runs/mcp` when that variable is unset.

The parser is the source of truth for flags. Use `worldfoundry-eval <command> [<subcommand>] --help` instead of copying flags from an unrelated model or benchmark recipe.

## Common workflows

### 1. Discover and check readiness

```bash
worldfoundry-eval zoo models
worldfoundry-eval zoo benchmarks --ready-now

worldfoundry-eval zoo model-show \
  --model-id <model-id> \
  --include-manifest \
  --json

worldfoundry-eval zoo benchmark-show \
  --benchmark-id <benchmark-id> \
  --include-spec \
  --json

# Checks the local cache; it does not download.
worldfoundry-eval zoo model-download \
  --model-id <model-id> \
  --check-local \
  --json
```

Use human tables while exploring. Add `--json` only when another program will consume the result.

### 2. Run one model directly

Inspect the resolved fields before allocating GPU resources:

```bash
worldfoundry-eval run self-forcing --model-status
worldfoundry-eval run self-forcing --print-config --json
```

Then provide only the overrides for this run:

```bash
worldfoundry-eval run self-forcing \
  --pipeline.prompt "A paper boat moving down a forest stream" \
  --pipeline.num-output-frames 33 \
  --pipeline.seed 7 \
  --pipeline.load.ckpt-path /path/to/self_forcing_dmd.pt \
  --runtime.device cuda \
  --output-dir tmp/self_forcing_run
```

Entries marked `runnable_runner` can execute directly. `runner_candidate` and `listed_only` entries remain inspectable but stop before model loading with a readiness explanation.

### 3. Plan or execute model × benchmark runs

Plan a potentially large matrix first:

```bash
worldfoundry-eval run \
  --all-benchmarks \
  --model <model-id> \
  --plan-only \
  --output-dir tmp/worldfoundry_plan \
  --json
```

Execute one cell after its assets and environment pass readiness checks:

```bash
worldfoundry-eval run \
  --benchmark <benchmark-id> \
  --model <model-id> \
  --mode official-run \
  --output-dir tmp/worldfoundry_run \
  --json
```

Repeat `--model` or `--benchmark` to create a matrix, or use `--suite <suite-id>`. Add `--resume` to reuse completed cells whose fingerprints still match.

### 4. Score existing artifacts or result rows

For a benchmark-owned artifact layout:

```bash
worldfoundry-eval score \
  --benchmark <benchmark-id> \
  --artifacts /path/to/generated-artifacts \
  --mode official-run \
  --output-dir tmp/score/<benchmark-id> \
  --plan-only \
  --json
```

Remove `--plan-only` after the prepared intent reports `ready: true`.

For already materialized WorldFoundry result rows:

```bash
worldfoundry-eval score \
  --results tmp/results.jsonl \
  --metric artifact_count \
  --metric required_artifacts_present \
  --required-artifact video \
  --output-dir tmp/score/results \
  --json
```

`evaluate` exposes the same deterministic core with lower-level request, task, runner, cache, and model-mode fields:

```bash
worldfoundry-eval evaluate \
  --results-path tmp/results.jsonl \
  --metric artifact_count \
  --required-artifact video \
  --output-dir tmp/worldfoundry_evaluate \
  --json
```

### 5. Import official-shaped benchmark results

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id <benchmark-id> \
  --mode official-validation \
  --official-results-path /path/to/official-results.json \
  --generated-artifact-dir /path/to/generated-artifacts \
  --output-dir tmp/<benchmark-id>/official-validation \
  --json
```

Benchmark-specific inputs such as data roots, score directories, prompt manifests, judge credentials, and metric subsets live on the corresponding [Benchmark Hub](/docs/evaluation/benchmark-hub) page.

For HarnessEval-W, `--result-model-id` selects a model without changing the official common-case cohort. See [HarnessEval-W](/docs/evaluation/benchmark-hub/harnesseval-w) for Python execution, cached scoring and model prerequisites.

## Benchmark modes

- **`official-run`** — Execute the benchmark runtime declared by the manifest against prepared artifacts. Use only when the required evaluator, data, checkpoints, and services are ready.
- **`official-validation`** — Import official-shaped results through the benchmark runner and record framework integration evidence.
- **`normalizer`** — Normalize provided result files into WorldFoundry artifacts without claiming that WorldFoundry executed the official scorer.
- **`contract`** — Exercise wiring/contracts only. It is not scoring evidence and cannot be leaderboard evidence.

<Callout type="warn" title="A successful command is not automatically a leaderboard result">
  Read `score_valid`, `leaderboard_valid`, `normalizer_only`, fidelity, and blocker fields in `scorecard.json`. Exit code 0 only means the requested CLI operation completed under its contract.
</Callout>

## Outputs and reproducibility

Global logging flags are accepted before or after the subcommand. They override
the corresponding environment values described in
[Environments → Logging](/docs/reference/environments#logging):

```bash
worldfoundry-eval --log-level DEBUG --log-json \
  --log-file tmp/worldfoundry.log zoo models --json
```

<div className="pi-kv-catalog not-prose">
  <section className="pi-kv-group">
    <h3>Flag</h3>
    <ul>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>--log-level</code>
          </div>
          <span className="pi-kv-default">
            <span>Env</span>
            WORLDFOUNDRY_LOG_LEVEL
          </span>
        </div>
        <p>One of <code>DEBUG</code>, <code>INFO</code>, <code>WARNING</code>, <code>ERROR</code>, or <code>CRITICAL</code>; default <code>INFO</code>.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>--log-file</code>
          </div>
          <span className="pi-kv-default">
            <span>Env</span>
            WORLDFOUNDRY_LOG_FILE
          </span>
        </div>
        <p>Optional rotating file sink.</p>
      </li>
      <li className="pi-kv-row">
        <div className="pi-kv-head">
          <div className="pi-kv-names">
            <code>--log-json</code>
          </div>
          <span className="pi-kv-default">
            <span>Env</span>
            WORLDFOUNDRY_LOG_JSON
          </span>
        </div>
        <p>JSON Lines for the file sink; console output stays text.</p>
      </li>
    </ul>
  </section>
</div>

Commands that own an output directory create a run id and, unless a file sink
was selected explicitly, write lifecycle events to a per-run `events.jsonl`.
This is normally below `<output-dir>/logs/<run-id>/`; commands that exclusively
claim their output directory use an isolated sibling log directory instead.
Framework-owned child processes inherit the structured
`WORLDFOUNDRY_LOG_CONTEXT` value.

Evaluation commands write durable files below `--output-dir`. Exact benchmark-owned extras vary, but the shared run surface uses these artifacts:

- **`run_manifest.json`** — Resolved run identity, status, counts, paths, cache evidence, and execution metadata.
- **`requests.jsonl` / `results.jsonl`** — Per-sample inputs and normalized generation results when that runner materializes ledgers.
- **`metrics/summary.json`** — Aggregate metrics and failure counts.
- **`artifacts.jsonl`** — Optional index of generated artifacts; disable with `--no-artifacts-index` where supported.
- **`scorecard.json`** — Metrics, fidelity, eligibility, leaderboard state, and blockers.
- **`summary.json` / `report.md`** — Compact reporting views when the selected runner emits them.
- **`suite_manifest.json` / `suite_report.md`** — Matrix identity and aggregate report for suite runs, plus one child directory per cell.

Keep the command, environment/profile identity, catalog revision, and output directory together when reproducing a score. See [Validation](/docs/reference/validation) for schema checks and release evidence.

For deterministic model outputs, configure the generation cache explicitly:

```bash
worldfoundry-eval run \
  --suite <suite-id> \
  --generation-cache-dir tmp/generation-cache \
  --generation-cache-mode read-write \
  --output-dir tmp/worldfoundry_suite \
  --json
```

The cache uses SQLite plus `audit.jsonl`; hits are recorded in the run manifest. Cache reuse does not override model or benchmark compatibility checks.

## Automation conventions

- Put `--json` on the leaf command: `worldfoundry-eval zoo models --json`.
- Repeat list flags instead of comma-joining unless that command explicitly accepts comma-separated values: for example, `--metric a --metric b` and `--model a --model b`.
- `KEY=VALUE` flags such as `--model-parameter`, `--model-runtime`, `--benchmark-parameter`, and `--env` parse JSON-compatible values where documented by `--help`.
- Use stable catalog IDs in scripts. Aliases are convenient interactively, but canonical IDs make stored commands easier to audit.
- Use `NO_COLOR=1` for plain redirected help, or `FORCE_COLOR=1` to keep ANSI color in a compatible terminal capture.
- Do not parse human tables. Use a command's `--json` output or the durable files under `--output-dir`.
- Add `--fail-on-sample-error` when partial sample failure must fail a batch or CI job.

### Exit status

- **`0`** — The requested operation completed. Inspect the output contract for score/leaderboard validity.
- **`1`** — A validation, metric, sample, prepared intent, or run result failed under a command that distinguishes domain failure.
- **`2`** — Invalid CLI input, missing required files/environment, preflight failure, or another execution/setup error.
- **`130`** — Interrupted with `Ctrl+C`.

Individual commands document stricter behavior in `--help`. Flags such as `--fail-on-issue`, `--fail-on-sample-error`, and `--fail-on-skipped` intentionally change when non-success exits are returned.

## Next steps

<Cards>
  <Card title="Quickstart" description="Go from a fresh clone to a reviewable first run." href="/docs/quickstart" />
  <Card title="Evaluation" description="Understand benchmark assets, official modes, and scorecards." href="/docs/evaluation" />
  <Card title="TUI" description="Build the same commands interactively in the terminal." href="/docs/guides/tui" />
  <Card title="Validation" description="Validate artifacts and release evidence." href="/docs/reference/validation" />
</Cards>