Show HN: Gemma 3 4B as a typed decision function in Rust (47 ms/decision)

Category: ai-ml

Tags: rust, llm-inference, candle, decision-making, shared-state, quantized-models

Score: 6.8/10 (Innovation: 7, Technical: 7, Documentation: 8, Utility: 5)

Gemma System One turns a quantized Gemma 3 4B model into a typed decision function in native Rust, using restricted logits over legal labels instead of open-ended generation. It explores shared-state inference optimizations (KV caching, batched isolated prefixes, packed question sheets) and measures the trade-off between latency and semantic drift, with a local API sidecar for agent routing. The project is interesting for its systems-level research into semantics-preserving inference and its practical benchmark against hosted decision models.

Target audience: AI/ML engineers, systems researchers, backend developers building agent routing layers

Repository: https://github.com/zozo123/gemma-to-jev · Rust

View on Hacker News