Show HN: Reflex – A GGUF-native Rust/CUDA engine for scale-to-zero LLM routing
Category: ai-ml
Tags: llm, inference, cuda, rust, gguf, serverless, cold-start
Score: 7.5/10 (Innovation: 7, Technical: 8, Documentation: 8, Utility: 7)
Reflex is a GGUF-native Rust/CUDA inference engine optimized for cold starts in serverless environments, achieving ~448 ms from process start to first result on an A10G GPU. It ships ahead-of-time compiled CUDA kernels and keeps weights quantized in GPU memory, targeting single-shot decision tasks like classification or tool calls with scale-to-zero routing.
Target audience: backend developers, devops, ml engineers deploying serverless LLM endpoints
Repository: https://console.runpod.io/hub/listing/lateos-ai/reflex · Rust · MIT · 63 stars
View on Hacker News