When it comes to building a streaming engine, choosing the right technology is crucial. At RudderStack, we decided to go with PostgreSQL over Apache Kafka, and it’s been a game-changer. We’ve been able to scale to 100,000 events per second, and it’s been a great decision for our customer data platform.
So, why did we choose PostgreSQL? For starters, we needed sophisticated error handling capabilities that involved blocking the queue for user-level failures, recording metadata about failures, maintaining event ordering per user, and updating event states for retries. Kafka’s immutable event model made this extremely difficult to implement.
We also needed superior debugging capabilities, which PostgreSQL provided with its SQL-like query capabilities. This allowed us to inspect queued events, update metadata, and force immediate retries – essential features for debugging and operational visibility.
Additionally, we needed multi-tenant scalability, which Kafka doesn’t provide. We needed separate queues per destination/customer combination to provide proper Quality of Service guarantees. PostgreSQL fit the bill perfectly.
Another key factor was management and operational simplicity. Kafka is complex to deploy and manage, and we didn’t want to ship and support a product where we weren’t experts in the underlying infrastructure. PostgreSQL, on the other hand, was a breeze to work with.
Lastly, we needed licensing flexibility. We wanted to release our entire codebase under an open-source license (AGPLv3), and Kafka’s licensing situation is complicated. PostgreSQL fit our licensing needs perfectly.
Have you ever had to make a similar decision? What was your thought process behind choosing one technology over another?