Show HN: Python script to evaluate your RAG pipeline against a golden set

Category: ai-ml

Tags: rag, evaluation, llm, python, cli

Score: 5.8/10 (Innovation: 4, Technical: 5, Documentation: 8, Utility: 6)

A Python script that evaluates RAG pipelines against a golden set using five metrics (retrieval recall, faithfulness, relevance, token F1, latency/cost) — no SaaS or API key required by default. It offers pluggable retriever/generator functions, terminal output, and optional LLM-as-judge. The README is thorough, with guidance on building golden sets and interpreting results, but the core metrics are relatively simple (token overlap, cosine similarity) and the space already has stronger frameworks like Ragas and DeepEval.

Target audience: AI/ML engineers and backend developers building RAG systems

Repository: https://github.com/infrixo-systems/rag-evaluation-starter · Python · MIT

View on Hacker News