Show HN: Evalship – find the flaws in your LLM evals

Category: ai-ml

Tags: llm-evals, statistical-testing, quality-assurance, ai-testing

Score: 4.3/10 (Innovation: 5, Technical: 4, Documentation: 3, Utility: 4)

Evalship is a quality-control tool for LLM evaluations that statistically detects whether differences in pass rates between eval variants are significant or due to chance. It pools test tries and sets a gap threshold, providing ranges and actionable insights to avoid overfitting or noise in LLM benchmarking.

Target audience: AI/ML engineers, data scientists, LLM application developers

Repository: https://evalship.com/

View on Hacker News