As a business PM transitioning into a data platform PM, I’ve been digging into the world of stream ingestion and compliance. One of the biggest challenges I’ve faced is handling different data types from multiple sources.
Usually, we’d modify data from sources to fit our needs, but for compliance purposes, that’s not an option. So, how do we ingest data from various sources with different data types? Is there a reference guide out there that can help me navigate this complex issue?
Another critical aspect is schema handling. What happens when there’s a schema change, like a new column or data type being added? Downstream ingestion breaks, and it’s a nightmare to fix. How do other data professionals handle these schema changes?
I’ve read the fundamentals of data engineering, but it didn’t quite cover these specific doubts. If anyone has experience with stream ingestion and compliance, I’d love to hear your insights.
As I’m deconstructing the product of my prospect company, I’m eager to learn from others who have tackled similar challenges. Share your knowledge, and let’s learn together!