Show HN: Jevcrawl – A web crawler for AI agents with no LLM in the crawl loop

Category: ai-ml

Tags: web-crawler, ai-agents, mcp, llm-cost-reduction, information-retrieval

Score: 7.5/10 (Innovation: 7, Technical: 7, Documentation: 9, Utility: 7)

Jevcrawl is a web crawler for AI agents that uses a small decision model (Jev) to score links and passages instead of an LLM, reducing cost and latency while improving answer-page discovery. It provides MCP tools for crawl, scrape, and map, with verbatim passage extraction and source citations. Benchmarks show it outperforms LLM-driven crawling and Crawl4AI on WebWalkerQA tasks at a fraction of the cost.

Target audience: AI engineers, backend developers, and agent builders

Repository: https://github.com/RayenMlayeh/jevcrawl · Python · Apache-2.0 · 80 stars

View on Hacker News