Show HN: Surprisal – a language identifier in ~250 lines of C. No ML,no training

Category: other

Tags: language-identification, compression, lz77, c, information-theory

Score: 5.8/10 (Innovation: 6, Technical: 6, Documentation: 7, Utility: 4)

Surprisal is a language identifier that uses a hand-rolled LZ77 compressor to measure how many bits it takes to encode a query given a reference text for each language. The least surprising language wins, with no machine learning or training involved. It's an interesting demonstration of the connection between compression and prediction, implemented compactly in around 250 lines of C.

Target audience: backend devs, researchers, hobbyists interested in compression or NLP

Repository: https://github.com/ahhyoushh/zip-surprisal · C · 5 stars

View on Hacker News