Show HN: DeepSeek's flagship got the same answers right and cost 9X per success

Category: devtools

Tags: llm, benchmark, cost-analysis, evaluation, multi-provider

Score: 6.5/10 (Innovation: 6, Technical: 6, Documentation: 8, Utility: 6)

A benchmark harness that measures the real cost per correct answer for LLMs across multiple providers, accounting for retries, reasoning tokens, and failed calls. It includes 16 code-checked tasks (plus a harder tier) with deterministic generated data, budget-capped runs, and resumable execution. It addresses a genuine gap in LLM evaluation by shifting focus from per-token price to cost-per-success.

Target audience: AI engineers, ML practitioners, and engineering managers evaluating LLM deployment costs

Repository: https://github.com/aicostbudget/llm-cost-per-task-benchmark · Python · MIT · 1 stars

View on Hacker News