Show HN: Spinf-serve – inference server for bulk scoring and decisions

Category: ai-ml

Tags: inference-server, classification, llm-serving, batch-invariance, gpu, prompt-packs

Score: 7.0/10 (Innovation: 7, Technical: 8, Documentation: 7, Utility: 6)

spinf-serve is an inference server that scores many questions against a single content read in one forward pass, returning calibrated probabilities per option rather than generated text. It targets bulk classification and decision-making workloads, with prompt packs, batch-invariant scoring, and an OpenAI-compatible API, and benchmarks favorably against vLLM for many-questions-per-document jobs.

Target audience: ML engineers, backend developers, data engineers working with bulk text classification and decision models

Repository: https://github.com/spinfinc/spinf-serve · Python · Apache-2.0 · 1 stars

View on Hacker News