Hey there, data engineers! I’m curious about how professional teams structure their codebases, especially when it comes to data engineering. Let’s say an organization has built an application – how do they organize their infrastructure, backend, and frontend code? Do they use a single monorepo or separate repositories for each component?
I’m particularly interested in learning about best practices for repository and folder structure, how CI/CD and deployments fit into the setup, and how team or organization size affects the approach. If you can, I’d love to see real-world examples of repository structures (folder trees, monorepo layouts, or links to public examples) and hear what’s worked or not worked for your team.
In terms of repository structure, I’ve seen some teams use a single monorepo for the entire application, while others break it down into separate repositories for infrastructure, backend, and frontend. Some teams also have a separate repository for data engineering work, such as data pipelines and data warehousing.
When it comes to folder structure, some common patterns include having separate folders for data, models, and scripts, as well as using a consistent naming convention for files and folders.
CI/CD and deployments can also vary widely depending on the team’s needs and infrastructure. Some teams use automated deployment scripts, while others rely on manual deployment processes.
I’ve also noticed that team or organization size can affect the approach to repository structure and management. Larger teams may require more formalized processes and stricter access controls, while smaller teams may be more agile and flexible in their approach.
If you have any insights or examples to share, I’d love to hear them!