Show HN: Run faster MoE inference on Mac with less RAM
Category: ai-ml
Tags: apple-silicon, moe, llm-inference, metal, gguf, local-ai
Score: 7.0/10 (Innovation: 7, Technical: 8, Documentation: 7, Utility: 6)
MoEShift is a C++ runtime that enables large Mixture-of-Experts LLMs (e.g., a 125B-parameter model) to run on Apple Silicon Macs using under 10 GB of peak memory by streaming experts from SSD with a lookahead scheduling scheme. It is interesting as a low-level systems/ML engineering project that meaningfully improves memory footprint and throughput over comparable tools like Strata, while offering an OpenAI-compatible local server.
Target audience: AI/ML engineers and macOS power users running local LLMs
Repository: https://github.com/davide-troiani/moeshift · C++ · MIT · 29 stars
View on Hacker News