LARYBench

LARYBench (Meituan LongCat) asks whether a video / world model's latent actions still contain real robot-action information. It is not a text-to-video beauty contest. The official pipeline has three stages over CALVIN, VLABench, AgiBot-Beta, and RoboCOIN:

• Extract — run a representation model over one metadata table, write latent-action CSVs, score extraction_coverage.
• Classify — probe discrete action classes (top1_accuracy, macro_f1, weighted_f1).
• Regress — probe continuous robot actions (val_seen_mse, val_unseen_mse) with --action-mode absolute|relative.

Paper: LARYBench. Project: meituan-longcat.github.io/LARYBench. Code / data: meituan-longcat/LARYBench and meituan-longcat/LARYBench (MIT).

WorldFoundry ships the stage drivers in tree (cli.py, extract.py, classification.py, regression.py, run_larybench_official_runner.py). Extraction partitions one metadata table across the selected GPUs and merges CSVs; classification runs one local process per GPU; regression uses Accelerate across CUDA_VISIBLE_DEVICES. Official classification_summary.json / best_result.json can be imported without loading datasets or checkpoints.

Dimensions

• Extract

  KindTask

  Run a representation model over a robot metadata table, write latent-action CSVs, and score extraction coverage.

• Classify

  KindTask

  Probe discrete action classes from extracted latents with top-1 accuracy and F1.

• Regress

  KindTask

  Probe continuous robot actions from extracted latents with seen/unseen validation MSE.

• CALVIN

  KindSplit

  Official CALVIN robot shards used by the LongCat pipeline.

• VLABench

  KindSplit

  Official VLABench robot shards used by the LongCat pipeline.

• AgiBot-Beta

  KindSplit

  Official AgiBot-Beta robot shards used by the LongCat pipeline.

• RoboCOIN

  KindSplit

  Official RoboCOIN robot shards used by the LongCat pipeline.

Data composition

• Size

  Meituan LongCat shards: robot videos, image pairs, trajectories, and metadata tables for CALVIN, VLABench, AgiBot-Beta, and RoboCOIN.

• Sources

  meituan-longcat/LARYBench · github.com

• Splits

  calvin · vlabench · agibot-beta · robocoin

• Modalities

  image · video · action · state

• License

  mit

• Notes

  WorldFoundry vendors extract, classify, and regress drivers in tree. Official media and representation checkpoints stay outside the package.

Metrics

Metric

• extraction_coverage

  DirectionHigher is better

  Fraction of metadata rows with successfully extracted latent actions.

  PrimaryLatent-action extraction coverage

• top1_accuracy

  DirectionHigher is better

  Top-1 accuracy of the classification probe over action classes.

• macro_f1

  DirectionHigher is better

  Macro-averaged F1 of the classification probe.

• weighted_f1

  DirectionHigher is better

  Support-weighted F1 of the classification probe.

• val_seen_mse

  DirectionLower is better

  Regression-probe action MSE on the seen validation split.

• val_unseen_mse

  DirectionLower is better

  Regression-probe action MSE on the unseen validation split.

Leaderboard

Official live board, embedded on this page. Not WorldFoundry-run scores. meituan-longcat.github.io/LARYBench.

How to run

Import an official classify or regress export:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id larybench \
  --mode official-validation \
  --official-results-path /path/to/classification_summary.json \
  --output-dir tmp/larybench/official-validation \
  --json

For regression, pass the official best_result.json instead; the runner infers the stage from the filename.

Full extract → classify → regress needs staged shards. Default import path when you omit --official-results-path is WORLDFOUNDRY_LARYBENCH_RESULTS_PATH.

export DATA_DIR=/path/to/larybench/shards
export LARY_METADATA_DIR=/path/to/larybench/metadata
export LARY_LA_DIR=/path/to/larybench/latent_actions
export MODEL_DIR=/path/to/representation_checkpoints

hf download meituan-longcat/LARYBench --repo-type dataset --local-dir "${DATA_DIR}"

${WORLDFOUNDRY_UNIFIED_PYTHON} \
  worldfoundry/evaluation/tasks/execution/runners/larybench/run_larybench_official_runner.py \
  --run-official \
  --stage extract \
  --dataset calvin \
  --model <representation_model> \
  --dataset-root "${DATA_DIR}" \
  --metadata-root "${LARY_METADATA_DIR}" \
  --latent-action-root "${LARY_LA_DIR}" \
  --model-root "${MODEL_DIR}" \
  --output-dir tmp/larybench/extract \
  --json

Classify needs --dim and --classes. Regress accepts --action-mode absolute|relative and optional --global-stats-json / --val-unseen-csv. Layout env vars: DATA_DIR, LARY_METADATA_DIR, LARY_LA_DIR, MODEL_DIR, optional WORLDFOUNDRY_LARYBENCH_ROOT.

In-tree pointers:

worldfoundry/data/benchmarks/tasks/external/larybench.yaml
worldfoundry/evaluation/tasks/execution/runners/larybench/

Outputs under --output-dir include scorecard.json (primary extraction_coverage), benchmark_contract.json, and raw_metric_table.jsonl.

← Benchmark Hub