Show HN: I built a hypervisor and client for inference on consumer compute

Category: ai-ml

Tags: distributed-inference, llm-serving, openai-compatible, peer-to-peer, rust

Score: 6.3/10 (Innovation: 6, Technical: 7, Documentation: 3, Utility: 6)

Scalattice is an OpenAI-compatible inference API that routes requests to a distributed network of consumer-hosted machines, with an open-source Rust agent for providers to earn money. It adds features like output vetting via replica responses, security tiers with split inference, and regional policy for data residency.

Target audience: AI developers, backend engineers, and hobbyists seeking cost-effective inference or wanting to monetize idle GPU capacity

Repository: https://scalattice.com/blog/openai-sdk-scalattice/

View on Hacker News