Show HN: Vramkit – measuring LLM weight and KV-cache compression on a MacBook

Category: ai-ml

Tags: llm-inference, model-compression, kv-cache, quantization, pytorch

Score: 7.0/10 (Innovation: 6, Technical: 7, Documentation: 8, Utility: 7)

vramkit provides measurable VRAM compression for LLM inference by optimizing both model weights and KV-cache. It introduces a lossless weight codec using a 2-D adaptive context model that beats general compressors, and demonstrates that KV-cache keys and values require different quantization axes—int8 per-channel keys achieve 2× compression at near-lossless perplexity. All benchmarks are reproducible on a MacBook, making it accessible without datacenter GPUs.

Target audience: ML engineers and researchers focused on efficient LLM inference

Repository: https://github.com/SRKRZ23/vramkit · Python · 4 stars

View on Hacker News