---
title: GenAI-Bench
description: "TIGER-Lab judge-alignment bench: pairwise accuracy vs human preferences on image generation, editing, and video. WorldFoundry imports labels; it does not invent a board."
---

GenAI-Bench (TIGER-Lab, University of Waterloo; released with GenAI-Arena) measures whether an **automatic judge matches human preference**, not whether a generator looks pretty. Each record pairs two outputs for the same prompt with a human label. WorldFoundry scores **pairwise accuracy** overall and on three tasks:

- **Image generation**
- **Image editing**
- **Video generation**

`genai_bench_average` is the mean of task accuracies when every task-level pair is present.

Paper: [GenAI-Bench](https://arxiv.org/abs/2406.04485). Code: [TIGER-AI-Lab/GenAI-Bench](https://github.com/TIGER-AI-Lab/GenAI-Bench). Dataset: [`TIGER-Lab/GenAI-Bench`](https://huggingface.co/datasets/TIGER-Lab/GenAI-Bench) (CC-BY-4.0).

WorldFoundry **normalizes official result rows**. A row must expose `task` (`image_generation` / `image_editing` / `video_generation`), a human label (`human_label` / `human_preference`), and a judge prediction (`prediction` / `judge_prediction`). It does not synthesize preference labels.

<BenchmarkPageSections id="genai-bench" locale="en" />

## How to run

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id genai-bench \
  --mode official-validation \
  --official-results-path <official_results.jsonl> \
  --generated-artifact-dir <generated_videos> \
  --output-dir tmp/genai-bench/official-validation \
  --json
```

`--preference-pairs-path` supplies the pair inventory. Default import path: `WORLDFOUNDRY_GENAI_BENCH_RESULTS_PATH`. Shared assets: `genai_bench_dataset_assets`, `t2v_metrics_source_assets`.

In-tree pointer: `worldfoundry/data/benchmarks/tasks/external/genai-bench.yaml`.

Outputs include `scorecard.json` (primary `genai_bench_average`), `benchmark_contract.json`, and `raw_metric_table.jsonl`.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)