Show HN: Llama.cpp fork with 2-4x multiGPU speed for MoE models bigger than VRAM

Category: ai-ml

Tags: llm-inference, moe, multi-gpu, expert-cache, llama-cpp, vram-optimization

Score: 7.8/10 (Innovation: 8, Technical: 9, Documentation: 7, Utility: 7)

A llama.cpp fork that accelerates multi-GPU inference for MoE models larger than total VRAM by treating free VRAM as a live expert cache, computing hot experts on GPUs while the CPU handles misses in parallel. It also introduces learned hot-expert persistence, dynamic micro-batch buffer lending, and in-place KV quantization fallback, with measured 2-4x speedups on several large MoE models. The project is technically deep and well-benchmarked, though it is a fork of a very active upstream.

Target audience: AI/ML engineers and hobbyists running large MoE models on consumer or multi-GPU setups with limited VRAM

Repository: https://github.com/neurall/llama.cpp · C++ · MIT · 12 stars

View on Hacker News