Show HN: Pith – a proxy that cuts LLM output tokens, and refuses when it can't

Category: infrastructure

Tags: llm, proxy, cost-optimization, python, api-gateway

Score: 7.0/10 (Innovation: 7, Technical: 7, Documentation: 8, Utility: 6)

Pith is a proxy that reduces LLM output token costs by automatically testing cheaper output profiles and only pinning them when a judge confirms they match baseline quality. It fingerprints routes, records samples, replays under candidate profiles, and refuses to change anything if the savings don't justify the sweep cost or if quality degrades. The refusal-first design and multiple integration points (standalone proxy, LiteLLM, Headroom, Portkey) make it a practical cost-optimization tool for API-based LLM applications.

Target audience: backend devs, devops, ml engineers

Repository: https://github.com/dex0shubham/pith · Python · MIT · 2 stars

View on Hacker News