Show HN: NullPrime – do AI models know when they can't know?
Category: ai-ml
Tags: llm-evaluation, calibration, uncertainty-quantification, benchmark, negative-control, primality
Score: 7.0/10 (Innovation: 7, Technical: 7, Documentation: 8, Utility: 6)
NullPrime is a benchmark dataset and protocol for testing whether AI models can recognize when a problem is unlearnable from its representation. It constructs prime/composite classification tasks where all small-prime divisibility signals are filtered out, making the label deterministic but unlearnable from features, and uses this as a negative control for LLM calibration and abstention. The project includes a pre-registered LLM study showing most models confidently answer at chance on these hard numbers.
Target audience: AI/ML researchers, LLM evaluation engineers, data scientists working on calibration and selective prediction
Repository: https://github.com/bmiedlar/nullprime · Python · MIT
View on Hacker News