Show HN: Web search engine built from the CommonCrawl corpus

Category: other

Tags: search-engine, commoncrawl, rust, tantivy, information-retrieval

Score: 6.0/10 (Innovation: 5, Technical: 7, Documentation: 3, Utility: 6)

A solo-built web search engine indexing 4 billion pages from CommonCrawl, using a Rust pipeline and Tantivy with BM25 and PageRank. It demonstrates that a significant search index can run on a single server rack, but details and docs are sparse, and it faces sustainability challenges.

Target audience: backend devs, data engineers, search engineers, researchers

Repository: https://search.xorsoft.dev/

View on Hacker News