Show HN: Rust crawler for SEO and content audits, with optional LLM review

Category: devtools

Tags: crawler, seo, website-audit, rust, markdown-export, ci-cd

Score: 6.8/10 (Innovation: 5, Technical: 7, Documentation: 8, Utility: 7)

SiteOne Crawler is a Rust-based website crawler and analyzer that audits pages for SEO, performance, and security, generates offline clones and Markdown exports, and offers an optional LLM review. It is notable for its single-binary cross-platform deployment, rich output formats, and CI/CD quality gate integration.

Target audience: web developers, devops, SEO specialists, content managers

Repository: https://crawler.siteone.io/ · Rust · MIT · 989 stars

View on Hacker News