Show HN: We quantized Qwen3.6-35B-A3B to 2-bit: 12.3 GB, 225 tok/s on a 4090
Category: ai-ml
Tags: quantization, moe, llm-inference, local-llm, openai-api
Score: 7.3/10 (Innovation: 7, Technical: 8, Documentation: 8, Utility: 7)
A 2-bit quantized version of the Qwen3.6-35B-A3B Mixture-of-Experts model, packaged with custom runtimes (SGLang-based and a Python-free ZML engine) for local serving via an OpenAI-compatible API. It achieves 12.3 GB on disk and 225 tok/s on an RTX 4090, making a 35B MoE model accessible on consumer GPUs. The detailed tuning guidance and verified configurations make it practical for local deployment.
Target audience: AI/ML engineers, backend developers, hobbyists running local LLMs
Repository: https://huggingface.co/EschaLabs/Qwen3.6-35B-A3B-Escha-W2
View on Hacker News