Show HN: Vramkit – lossless VRAM compression for LLM inference, on a MacBook

Category: ai-ml

Tags: llm-inference, vram-compression, kv-cache, quantization, pytorch

Score: 6.8/10 (Innovation: 6, Technical: 7, Documentation: 8, Utility: 6)

vramkit provides lossless weight compression via a custom 2D context range coder that beats zstd/xz on quantized LLM weights, and a KV-cache compression scheme emphasizing per-channel quantization for keys to achieve near-lossless 2x reduction. Benchmarks are runnable on a MacBook and validated with real end-task perplexity, with honest reporting of a previous flawed methodology.

Target audience: ML engineers, LLM inference engineers, GPU/ROCm developers

Repository: https://github.com/SRKRZ23/vramkit · Python · 4 stars

View on Hacker News