Show HN: Real-Time Bluesky Post Firehose in < 256MB

Category: infrastructure

Tags: stream-processing, sql, duckdb, kafka, go

Score: 7.5/10 (Innovation: 7, Technical: 8, Documentation: 8, Utility: 7)

SQLFlow is a lightweight stream processing engine that uses DuckDB to run SQL over streaming data from Kafka, WebSockets, or webhooks, and sinks results to Kafka, ClickHouse, Iceberg, or other DuckDB-compatible destinations. It stands out by delivering high throughput (~900k msgs/sec) in a single Go binary using under 256MB of memory, making it a compelling alternative to heavier systems like Flink for SQL-centric stream processing.

Target audience: data engineers, backend developers, devops

Repository: https://turbolytics.io/demos/bluesky/ · Go · MIT · 797 stars

View on Hacker News