Have you ever wondered how to turn a classic eCommerce dataset into a real-time data generator? That’s exactly what we did with theLook eCommerce dataset, and I’m excited to share our journey with you.
## The Challenge
The theLook eCommerce dataset is a well-known resource for data engineers and analysts. However, it was initially designed for batch workloads, making it less suitable for real-time analytics. We saw an opportunity to re-engineer it into a real-time data generator that streams simulated user activity directly into PostgreSQL.
## The Solution
By leveraging Change Data Capture (CDC) with Debezium and Kafka, we created a powerful pipeline that can handle real-time data processing. This setup allows us to test real-time analytics on a realistic schema, experiment with event-driven architectures, and push the boundaries of what’s possible with eCommerce data.
## The Benefits
With our real-time data generator, you can:
– Build CDC pipelines with Debezium + Kafka
– Test real-time analytics on a realistic schema
– Experiment with event-driven architectures
## Get Involved
Want to explore this project further? Check out our GitHub repo, where you can find the code and instructions to get started. We’re excited to hear how you might extend or build upon this project!
[Further reading: theLook eCommerce dataset on Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/bigquery-public-data/thelook-ecommerce)
[Check out the GitHub repo](https://github.com/factorhouse/examples/tree/main/projects/thelook-ecomm-cdc)