Airflow Python Developer

Role Overview

Sutherland is seeking an experienced Python Data Engineer to design, build, and orchestrate scalable ETL/ELT workflows using Apache Airflow on cloud platforms like AWS and Snowflake. You will play a key role in automating data processes, ensuring high availability of production pipelines, and collaborating with cross-functional teams to deliver robust data solutions.

Key Responsibilities

• Create and maintain robust data pipelines leveraging Python, Apache Airflow (DAGs), and cloud services such as AWS S3, Glue, Lambda, Athena, or Snowflake.
• Design custom operators, sensors, hooks within Airflow to optimize scheduling and monitoring of complex workflows.
• Develop efficient SQL queries for data transformation in ELT solutions using Snowflake, Redshift, BigQuery, Databricks, or similar warehouses.
• Implement automated ingestion, validation, and quality checks to ensure data integrity across systems.
• Troubleshoot pipeline failures proactively to maintain high availability of production workflows.

Qualifications

To be successful in this role, you should have:

• 2+ years of experience developing Python scripts for automation and data engineering tasks.
• 1+ year hands-on expertise with Apache Airflow for DAG design, orchestration, and scheduling.
• Familiarity with AWS services (S3, Glue, Lambda) or equivalent cloud technologies like Azure/GCP.
• Advanced SQL skills focusing on query optimization and performance tuning.
• Solid understanding of ETL/ELT concepts, data quality frameworks, and validation strategies.
• Basic knowledge of Snowflake or other major cloud warehouses (Redshift, BigQuery, Databricks).
• Experience with version control systems like Git and familiarity with CI-CD pipelines is a plus.