Show HN: Modelship – an open source ray backed model cluster

Category: ai-ml

Tags: model-serving, openai-compatible, ray, inference, llm-ops

Score: 7.3/10 (Innovation: 6, Technical: 8, Documentation: 8, Utility: 7)

Modelship is an open-source, Ray-backed model serving cluster that exposes an OpenAI-compatible API surface, including the Responses API with tool calling, server-side MCP execution, and shared conversation state. It aims to let teams self-host agentic backends with multi-model, multi-GPU, and CPU support while preserving client compatibility and zero-downtime model updates.

Target audience: ML engineers, backend devs, and platform/DevOps teams deploying self-hosted LLM inference

Repository: https://github.com/modelship-ai/modelship · Python · Apache-2.0 · 54 stars

View on Hacker News