Show HN: 2-bit Qwen3.6-35B-A3B with ~100% FP8 quality retention
Category: ai-ml
Tags: quantization, mixture-of-experts, llm-inference, local-llm, openai-api, model-serving
Score: 6.7/10 (Innovation: 6, Technical: 8, Documentation: 8, Utility: 6)
A 2-bit quantized build of Qwen3.6-35B-A3B (Mixture-of-Experts with 256 experts) packaged with an OpenAI-compatible runtime, enabling the model to run on a single 24 GB or 16 GB consumer GPU with reportedly near-FP8 quality. The project pairs aggressive mixed-precision quantization (2/3-bit experts, int8 dense layers) with two serving engines (SGLang and a dependency-free ZML runtime) and detailed per-GPU tuning recipes.
Target audience: AI engineers, ML practitioners, and local LLM enthusiasts running models on consumer GPUs
Repository: https://huggingface.co/EschaLabs/Qwen3.6-35B-A3B-Escha-W2
View on Hacker News