Show HN: Sturnus, a latency-based LLM load balancer
Category: infrastructure
Tags: llm, load-balancer, latency-routing, sidecar, openai-api, rust
Score: 7.0/10 (Innovation: 6, Technical: 7, Documentation: 9, Utility: 6)
Sturnus is a lightweight Rust sidecar that provides an OpenAI-compatible API and automatically routes LLM requests to the fastest healthy provider using latency-aware EWMA scoring with error feedback. It stands out for its zero-infrastructure, single-binary design, transparent passthrough, session affinity, and built-in observability, making latency-based routing accessible without a full LLMOps platform.
Target audience: backend and platform engineers deploying LLM applications in production
Repository: https://github.com/sturnus-dev/sturnus · Rust · MIT · 12 stars
View on Hacker News