Show HN: InferPilot – the vLLM doctor that tells you when fp8 won't help

Category: observability

Tags: vllm, inference-optimization, llm-ops, prometheus, observability, gpu-performance

Score: 7.0/10 (Innovation: 7, Technical: 7, Documentation: 8, Utility: 6)

InferPilot is a diagnostic tool for vLLM inference servers that reads live Prometheus metrics to determine whether the server is KV-cache pressured or compute-bound, then recommends whether levers like fp8 KV cache are worth testing. It emphasizes conservative, evidence-based decision-making and abstains when evidence is insufficient, with a full pipeline for controlled experiments and quality gates.

Target audience: MLOps engineers, backend developers running LLM inference, DevOps

Repository: https://github.com/poojithdevan4D/InferPilot · Python · MIT · 3 stars

View on Hacker News