Show HN: Resin, fast native scheduler for LLM inference engines

Category: ai-ml

Tags: llm-inference, scheduler, rust, cuda, kv-cache

Score: 7.8/10 (Innovation: 8, Technical: 9, Documentation: 8, Utility: 6)

Resin provides a Rust-based scheduler and KV cache for LLM inference engines, separating scheduling logic from model-specific forward pass kernels in Python. It supports advanced features like speculative decoding, parallelisms, CUDA graphs, and an HTTP serve layer, allowing rapid development of custom inference engines with competitive performance.

Target audience: ML systems engineers, AI infrastructure developers, performance engineers

Repository: https://github.com/ekzhang/resin · Python · Apache-2.0 · 207 stars

View on Hacker News