Show HN: Find duplicates in 5 lines of Python (offline, lightweight, fast, Rust)

Category: library

Tags: duplicate-detection, entity-resolution, fuzzy-matching, embeddings, offline, rust, python

Score: 7.3/10 (Innovation: 6, Technical: 8, Documentation: 8, Utility: 7)

Univo is a Rust-backed Python library for duplicate detection and entity resolution, combining fuzzy string matching, local embeddings, and field-level rules. It offers an offline, lightweight solution with schema-based record comparison, semantic text modes using Model2Vec, and fast incremental checking, making it practical for data cleaning, deduplication, and record linkage without training or external APIs.

Target audience: data engineers, data scientists, backend developers

Repository: https://github.com/leopoldch/univo · Rust · MIT · 1 stars

View on Hacker News