Show HN: Reverse engineering the ANE to run Gemma 4 26B at 100 Tok/s
Category: ai-ml
Tags: apple-neural-engine, llm-inference, speculative-decoding, apple-silicon, swift
Score: 7.5/10 (Innovation: 8, Technical: 9, Documentation: 7, Utility: 6)
ANE Fieldfare is a Swift-based inference engine that runs Gemma 4 26B-A4B on Apple Silicon by leveraging the Neural Engine for prefill, GPU-based speculative decoding with a DFlash drafter, and memory-mapped expert weights cached by the OS file cache. It achieves up to 100 tok/s on a 16GB Mac with under 1GB RSS, significantly outperforming llama.cpp and Turbo Fieldfare.
Target audience: AI/ML engineers, Apple Silicon developers, local LLM enthusiasts
Repository: https://github.com/zw5/ane-fieldfare · Swift · Apache-2.0 · 6 stars
View on Hacker News