As a seasoned SQL developer transitioning to data engineering, I’ve realized that my current toolkit is no longer cutting-edge. I’m using Python to ETL data into Postgres, and relying on Windows tasks with a custom .NET orchestrator. But I know I need to upskill to stay competitive in the job market.
The question is, where to start? With so many modern tools and technologies emerging, it’s overwhelming to determine which ones are essential for a modern data engineer. Do I need to learn dbt, Snowflake, or Airflow? Should I explore newer players like Dagster, Mage, or Prefect?
In this post, I’ll explore the minimum tools required to be a modern data engineer. We’ll dive into the must-haves, the nice-to-haves, and the emerging trends that will help you stay ahead of the curve.
Whether you’re looking to upskill or reskill, this post will provide a roadmap for navigating the complex landscape of data engineering tools and technologies.