Show HN: Self-bench – benchmark coding agents on real-world software

Category: ai-ml

Tags: benchmarking, coding-agents, llm-evaluation, pull-requests, typeScript

Score: 7.5/10 (Innovation: 8, Technical: 8, Documentation: 7, Utility: 7)

SelfBench generates private coding-agent benchmarks from a repository's own merged pull requests by reconstructing tasks, writing hidden tests, and validating them through a multi-gate pipeline. It runs agents and models on these tasks to plot accuracy versus cost with a Pareto frontier, filling a real gap between generic public benchmarks and repo-specific performance.

Target audience: AI/ML engineers, developer productivity teams, and engineering managers evaluating coding agents for their own codebases

Repository: https://github.com/mupt-ai/self-bench · TypeScript · MIT · 33 stars

View on Hacker News