Show HN: Zero-shot classifier on scale-to-zero GPUs, one forward pass per call
Category: ai-ml
Tags: inference-engine, gguf, cuda, rust, cold-start, serverless, gpu
Score: 7.8/10 (Innovation: 7, Technical: 9, Documentation: 8, Utility: 7)
Reflex is a GGUF-native Rust and CUDA inference engine optimized for cold starts, compiling all CUDA kernels ahead of time to achieve ~448ms first result on scale-to-zero GPUs. It targets single-tenant, one-shot inference (classification, extraction) where startup latency dominates cost, and includes a serverless sidecar for OpenAI-compatible HTTP.
Target audience: machine learning engineers, backend developers deploying serverless AI
Repository: https://console.runpod.io/hub/listing/lateos-ai/reflex · Rust · MIT · 63 stars
View on Hacker News