As a non-technical person leading a team in the Democratic Republic of Congo, I knew I had to find a way to streamline our data management process. We were stuck in a cycle of manual data extraction from various systems, wrestling with CSVs and PDFs in Excel, and struggling to make sense of it all.
I decided to take the first step towards building a proper data pipeline, knowing that it would open up opportunities for automation, reporting, and even AI/ML applications. But, I needed guidance from the experts.
The Vision
I envisioned a data stack that would integrate our disparate systems, automate reporting, and provide insights for procurement, sales forecasting, and more. I was comfortable with basic SQL and Python, but I knew I needed help to bring this vision to life.
The Questions
I had several questions that I needed answers to:
- Is my proposed data stack a good starting point for a small, non-technical team?
- What should I prioritize in the first phase, and what can wait?
- Should I DIY or bring in a freelancer, and what are the budget and time implications?
The Takeaways
After seeking feedback from the data engineering community, I learned that my initial vision was a good starting point. Here are some key takeaways:
- Start small: Focus on automating reporting and building a solid data foundation before diving into AI/ML applications.
- Prioritize integration: Connect your systems and automate data extraction to reduce manual labor.
- DIY or freelancer?: Depending on your team’s size and resources, you may want to DIY or bring in a freelancer. Upwork, Fiverr, and small boutique firms can be good options.
- Budget and time: Be prepared to invest time and resources in building your data stack. The cost will depend on the scope and complexity of your project.
The Next Steps
If you’re in a similar situation, don’t be afraid to seek guidance from the community. With a clear vision, a willingness to learn, and the right support, you can build a data stack that empowers your non-technical team to make data-driven decisions.
Further reading: Data Engineering for Non-Technical Teams