Show HN: Scrubbed – fast native web-data cleaning for LLM training

Category: devtools

Tags: web-scraping, data-cleaning, llm-training, text-processing, pii-detection, deduplication, native-binary

Score: 7.3/10 (Innovation: 6, Technical: 8, Documentation: 9, Utility: 6)

Scrubbed is a native binary written in D that combines encoding repair, HTML main-content extraction, PII scanning, language identification, and deduplication into a single-pass pipeline for cleaning web data for LLM training. It aims to replace a chain of Python tools with a faster, zero-copy, multi-threaded alternative, and includes verified correctness against reference tools along with detailed benchmarks and documentation.

Target audience: data engineers and ML engineers building LLM training pipelines

Repository: https://schancel.github.io/scrubbed/ · D · MIT · 3 stars

View on Hacker News