Show HN: Has Anthropic been nerfing their models without disclosure

Category: ai-ml

Tags: benchmarking, llm-evaluation, model-drift, reproducibility, statistics

Score: 7.5/10 (Innovation: 7, Technical: 7, Documentation: 9, Utility: 7)

livenerf is a pre-registered, long-running benchmark that monitors whether Anthropic's Claude Opus 5.5 quietly degrades after launch by running a frozen panel of 78 questions daily and measuring accuracy and token drift. It uses the Inspect eval framework and rigorous statistics (paired per-item differences, clustered standard errors) to distinguish real regressions from noise, with full logging and a public results table.

Target audience: AI researchers, ML engineers, and technically-minded users concerned with model reliability and transparency

Repository: https://github.com/ninjahawk/livenerf · Python · MIT · 1552 stars

View on Hacker News