Show HN: GridCore – a scheduler that lets several local LLMs share one GPU

Category: infrastructure

Tags: gpu-scheduling, llm-inference, llama-cpp, vram-management, local-ai

Score: 7.5/10 (Innovation: 7, Technical: 8, Documentation: 8, Utility: 7)

GridCore is a GPU workload scheduler for local LLM inference that manages VRAM, model residency, and request prioritization across multiple llama.cpp servers. It introduces OS-like scheduling with interactive/background/batch classes, VRAM-aware admission, and self-correcting eviction, addressing a real pain point for developers running multiple local models on a single GPU.

Target audience: AI/ML engineers and hobbyists running local LLMs on single-GPU machines

Repository: https://blokhin.us/notes/gridcore-gpu-scheduler/ · Go · MIT · 6 stars

View on Hacker News