Show HN: Agent-trials – a smoke-test runner for coding agent CLIs

Category: devtools

Tags: agent-cli, benchmarking, smoke-test, developer-tools, typescript

Score: 6.3/10 (Innovation: 6, Technical: 6, Documentation: 8, Utility: 5)

agent-trials is a TypeScript-based smoke-test runner that evaluates coding agent CLIs (like Claude, Codex, Gemini) on user-defined tasks, producing a scoreboard of pass/fail, time, and cost. It emphasizes execution safety by withholding credentials, skipping secret files and symlinks, and enforcing timeouts, though it explicitly warns it is not a sandbox. The project is interesting for its practical approach to benchmarking agents on your own codebase rather than generic benchmarks.

Target audience: developers and engineering teams evaluating coding agents

Repository: https://github.com/princejain756/agent-trials · TypeScript · MIT · 2 stars

View on Hacker News