Show HN: OmnisBench, a re-gradable, open LLM routing benchmark on fresh tasks

Category: ai-ml

Tags: llm-routing, benchmark, llm-evaluation, contamination-detection, cost-optimization

Score: 6.3/10 (Innovation: 6, Technical: 6, Documentation: 8, Utility: 5)

OmnisBench is an open, re-gradable benchmark for LLM routing policies, measuring how close a router gets to the ideal quality-per-dollar frontier on fresh, uncontaminated tasks. It distinguishes itself from RouterBench by emphasizing live, cost-current, offline-verifiable grading and explicit contamination accounting via fresh splits and perturbation probes.

Target audience: AI/ML engineers and researchers working on LLM routing, model selection, or evaluation infrastructure

Repository: https://github.com/Fortitude-Group/OmnisBench · Python · Apache-2.0

View on Hacker News