Show HN: A layer on top llama.cpp enabling big MoE streaming on phone and laptop

Category: ai-ml

Tags: moe, llama-cpp, edge-computing, model-streaming, cross-platform

Score: 8.5/10 (Innovation: 8, Technical: 9, Documentation: 9, Utility: 8)

BigMoeOnEdge is a C++ layer on top of llama.cpp that enables running large Mixture-of-Experts models on devices with limited memory by streaming expert weights from flash storage. It includes an automatic planner that measures hardware and model characteristics to optimize cache and device placement, and provides native apps for mobile and desktop with an OpenAI-compatible server. The project demonstrates impressive cross-platform performance, running models several times larger than available RAM on phones and laptops.

Target audience: AI/ML engineers, mobile developers, edge computing enthusiasts

Repository: https://github.com/Helldez/BigMoeOnEdge · C++ · Apache-2.0 · 646 stars

View on Hacker News