Show HN: Run Jev-style models locally on Mac with 0.74 GB RAM

Category: ai-ml

Tags: apple-silicon, local-inference, memory-optimization, typed-decisions, pytorch-mps

Score: 6.0/10 (Innovation: 5, Technical: 7, Documentation: 8, Utility: 4)

Laya MPS is a local inference server for a specialized typed-decision model, optimized for Apple Silicon Macs with aggressive memory-saving techniques like streaming layers and embeddings from disk. It achieves low RAM usage (0.74 GiB) and fast latency (32 ms) by carefully managing PyTorch MPS allocations, making it an interesting engineering study in memory optimization for edge inference.

Target audience: backend devs, ML engineers, Mac developers

Repository: https://github.com/afshinm/laya-mps · Python · MIT · 22 stars

View on Hacker News