Small Kafka: Tansu + SQLite on a free t3.micro (AWS Free Tier)
Achieving 7000 msg/s (~6.7MB/s) with just 27MB of memory on a free t3.micro EC2 in AWS
I’m Peter, the founder of tansu.io, we develop, license and support an Apache Kafka® compatible broker with SQLite, PostgreSQL and S3 storage engines with Apache Parquet, Apache Iceberg and Delta Lake.
Tansu SQLite: 48MB/sec, 114ms max latency with 20MB RSS
Using flame graphs to remove a hot path, stop copying data and switching to a fast CRC32
Tuning Tansu: 600,000 record/s with 13MB of RAM
Writing 585 MB/s with just 13MB of memory usage.
SQLite storage for Tansu
Using SQLite as a storage engine with Tansu, a Kafka compatible streaming platform, producing and consuming Protobuf messages using generated test data.