---
title: CALVIN
description: "Long-horizon language-conditioned tabletop (RA-L 2022). Official protocol is success at chain lengths 1–5 after training on A/B/C and testing on D."
---

CALVIN (Composing Actions from Language and Vision) is the University of Freiburg's long-horizon, language-conditioned manipulation benchmark (IEEE RA-L 2022). A policy in a **PyBullet** tabletop must chain up to **five** language instructions. Four environment configurations **A/B/C/D** test zero-shot transfer: the signature split trains on A/B/C and evaluates on unseen **D**. The paper reports success at each chain length and the **average number of completed instructions**, not a single binary episode bit.

Paper: [CALVIN](https://ieeexplore.ieee.org/document/9788026) (also discussed at [calvin.cs.uni-freiburg.de](http://calvin.cs.uni-freiburg.de/)). Code: [mees/calvin](https://github.com/mees/calvin) (MIT).

WorldFoundry is an **official-results importer**. You run the upstream sequence evaluator, then normalize the export. Simulator, dataset roots, and policy checkpoints follow the official CALVIN docs. Repo mirror: `cache/vla_va_wam/repos/mees--calvin`.

<BenchmarkPageSections id="calvin" locale="en" />

## How to run

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id calvin \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --output-dir tmp/calvin/official-validation \
  --json
```

Default import path: `WORLDFOUNDRY_CALVIN_RESULTS_PATH`. Manifest: `worldfoundry/data/benchmarks/tasks/external/calvin.yaml`. Outputs: `scorecard.json` (primary `success_rate`; also `sequence_success`, `completion`) and `raw_results.jsonl`.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)