HarnessEval-W

HarnessEval-W evaluates exploratory, intentional and physical transitions, drift resistance, return/revisit consistency, and offscreen evolution. The public dataset is reported as 100 cases. WorldFoundry pins evaluator revision ed4ccc6486b8271723ee8baea60d89b32d0a7518.

Each published case score is 50% selected-core score and 50% observation quality. overall_macro averages the six family means equally, using exact common cases across all models in the input. Missing families leave the overall score unavailable. Use --result-model-id when importing a report containing multiple models; selection preserves the original comparison cohort.

CPU validation and result import

The synthetic fixture checks schemas and aggregation without data downloads or models:

python -m worldfoundry.evaluation.tasks.execution.runners.harnesseval_w.run_harnesseval_w \
  --run-fixture --strict --output-dir tmp/harnesseval-fixture --json

Import an official harnesseval.report JSON, a case-score JSON/list, or an evaluation directory containing rollout_scores/<model>/<family>/*.scores.json:

worldfoundry-eval zoo benchmark-run --benchmark-id harnesseval-w \
  --mode official-validation --official-results-path /path/to/evaluation \
  --result-model-id my-model --output-dir tmp/harnesseval-import --json

Outputs include scorecard.json, raw_metric_table.jsonl, per_case_metrics.jsonl, official_report.json and benchmark_contract.json. A summary-only import has no per-case evidence. Fixtures and imported results never claim leaderboard validity.

Evaluate generated videos

Keep generation in WorldFoundry. Supply the official case manifest, initial observations, unmodified case skill plans and generated artifacts:

generation/manifest.json
generation/outputs/<axis>/<family>/<model>/<case>/output.mp4
generation/outputs/<axis>/<family>/<model>/<case>/metadata.json
plans/<family>/<case>.skill_plan.json

axis is transition_correctness or world_persistence. Metadata declares case_id, model_id, taxonomy and output_video (relative to metadata or absolute). Initial-observation paths resolve against --assets-root. Backend JSON uses the official configuration schema. Use api_key_env for judge credentials. Configure weights_root outside the package source tree. HPSv3 uses registered WorldFoundry HPSv3 assets; AMT/RAFT/CLIP/aesthetic weights reuse staged base-model assets when absent from the configured directory.

python -m worldfoundry.evaluation.tasks.execution.runners.harnesseval_w.run_harnesseval_w \
  --run-official --generated-artifact-dir /path/to/generation \
  --manifest /path/to/manifest.json --plan-root /path/to/plans \
  --assets-root /path/to/dataset --model-id my-model \
  --backend-config /path/to/backend_config.json \
  --output-dir tmp/harnesseval-run --strict --json

Add --dry-run to audit inputs and plan complete skill sets without inference. --limit N selects N cases after auditing the supplied manifest. For existing official metric bundles, use --score-cached --metric-cache-root /path/to/metric_cache; this checks cached evidence and executes official scoring without loading models. Each skill, including each staged drift component, runs in a bounded Python process. --timeout controls each worker; logs and audits are written under the output directory.

Full inference needs CUDA, ffmpeg/ffprobe, model dependencies, compatible weights and a VLM judge. CLIP, RAFT, AMT, HPSv3, Depth Anything v1, DINOv2, MegaSAM and UniDepth reuse worldfoundry/base_models. UniDepth uses the existing geometry-prior loader and registered WORLDFOUNDRY_UNIDEPTH_V2_VITL14_MODEL_DIR (containing config.json and model.safetensors). MegaSAM tracking and UniDepth inference sources are included in their canonical base-model locations. UniDepth uses the upstream compatibility decoder matching the pinned HF snapshot; strict loading and GPU depth/points/intrinsics inference have been verified. Its Nyström attention requires xformers.components.attention (tested with xformers==0.0.28.post3). Build MegaSAM's two CUDA extensions from worldfoundry/base_models/three_dimensions/slam/mega_sam_runtime/base with python -m pip install --no-build-isolation -e . using a compiler compatible with PyTorch. Both WBench and HarnessEval-W call the same resident Python pipeline. No shell launchers or generator copies are included. The full 100-case GPU/API evaluation and model parity remain unverified.

For the unified official-run entrypoint, supply the extra inputs through WORLDFOUNDRY_HARNESSEVAL_MANIFEST, WORLDFOUNDRY_HARNESSEVAL_PLAN_ROOT, WORLDFOUNDRY_HARNESSEVAL_MODEL_ID and WORLDFOUNDRY_HARNESSEVAL_BACKEND_CONFIG. Set WORLDFOUNDRY_HARNESSEVAL_PYTHON only when skill workers require a different prepared interpreter. Upstream declares Apache-2.0 in its README but includes no standalone LICENSE file at the pinned revision.