Show HN: DoubtBench – does Jev know when humans disagree?

Category: ai-ml

Tags: benchmark, uncertainty, calibration, human-annotation, evaluation

Score: 6.5/10 (Innovation: 7, Technical: 6, Documentation: 7, Utility: 6)

DoubtBench is a benchmark that evaluates decision models not just on accuracy but on whether their uncertainty aligns with human disagreement, using NVIDIA's HelpSteer2 dataset. It provides a leaderboard and tooling to run models like Jev and Claude, revealing that no current model hesitates where humans disagree, which is crucial for human-in-the-loop systems.

Target audience: AI researchers, ML engineers, data scientists

Repository: https://github.com/deeplearningguy/doubtbench · Python · MIT · 3 stars

View on Hacker News