Show HN: Narwhal – LLM serving that moves GPUs between prefill/decode in seconds

Category: ai-ml

Tags: llm-inference, gpu-scheduling, disaggregated-serving, kv-cache, vllm, sglang

Score: 7.5/10 (Innovation: 8, Technical: 8, Documentation: 8, Utility: 6)

Narwhal is a disaggregated LLM inference framework that dynamically hot-swaps prefill and decode roles across a fixed GPU fleet without reloading model weights, using NIXL for KV transfer. It implements scheduling algorithms from the Arrow paper and provides fault tolerance, observability, and benchmarking tools for single-GPU or multi-node deployments. This is interesting because it tackles the resource inefficiency of static prefill/decode splits in LLM serving.

Target audience: ML infrastructure engineers, backend developers deploying LLMs

Repository: https://github.com/athrael-soju/Narwhal · Python · Apache-2.0 · 8 stars

View on Hacker News