Show HN: Capy-flash – a 125B MoE model on two 16 GB consumer GPUs
Category: ai-ml
Tags: inference-engine, mixture-of-experts, gpu, quantization, tensor-parallelism, llm-serving
Score: 8.0/10 (Innovation: 8, Technical: 9, Documentation: 8, Utility: 7)
Capy-flash is a C++ inference engine that serves a 125B MoE model (Qwen3.8-Flash-Next) in NVFP4 on two 16 GB consumer GPUs by offloading routed experts to CPU RAM and using tensor parallelism over PCIe without peer-to-peer. It includes agent-oriented features like context caching, expert residency tracking, and MTP speculative decoding, with measured performance competitive with larger setups.
Target audience: AI engineers, ML researchers, and hobbyists running large language models on consumer hardware
Repository: https://github.com/ValerioDolci/capy-flash · C++ · Apache-2.0 · 67 stars
View on Hacker News