Show HN: Benchmark: AI doesn't find bugs unless you tell it what's wrong

Category: ai-ml

Tags: benchmark, ai-agents, bug-fixing, code-repair, evaluation

Score: 6.5/10 (Innovation: 7, Technical: 6, Documentation: 7, Utility: 6)

SWE-sweep is a benchmark for evaluating AI coding agents on autonomously discovering and fixing multiple concurrent bugs in large open-source repositories without any hints about bug location or type. It constructs tasks from real issue-PR pairs and measures agents' ability to explore codebases, implement fixes, and avoid regressions, filling a gap in open-ended software engineering evaluation.

Target audience: AI researchers, machine learning engineers, software engineering researchers

Repository: https://swesweep.com/ · Python · NOASSERTION · 177 stars

View on Hacker News