Show HN: Building a Magic the Gathering Benchmark

Category: ai-ml

Tags: llm-evaluation, game-benchmark, mtg, agent-harness, reasoning

Score: 6.3/10 (Innovation: 7, Technical: 6, Documentation: 6, Utility: 6)

This project builds a benchmark for evaluating LLMs by having them play Magic: The Gathering through a custom harness on the open-source Forge rules engine. It addresses the challenge of creating repeatable, differentiating, and trustworthy evals by using deterministic puzzles and a polished tool-call interface that gives models full game state and simulation abilities.

Target audience: AI researchers, LLM evaluation engineers, game AI developers

Repository: https://folds.ink/posts/mtg-benchmark

View on Hacker News