Show HN: Reflex – a GGUF/CUDA inference engine tuned for cold-start latency

Category: ai-ml

Tags: inference-engine, cuda, gguf, cold-start, serverless

Score: 7.3/10 (Innovation: 7, Technical: 8, Documentation: 8, Utility: 6)

Reflex is a Rust and CUDA inference engine for GGUF models that optimizes cold-start latency by precompiling kernels ahead of time, targeting serverless GPU workloads. It achieves ~448ms cold start on an A10G for small models and supports multiple architectures, with extensive benchmarks comparing against llama.cpp, Ollama, and vLLM.

Target audience: ML engineers, backend developers, devops deploying serverless GPU inference

Repository: https://github.com/lateos-ai/reflex · Rust · MIT · 63 stars

View on Hacker News