Senior Data Engineer

About the Role

We are seeking an experienced Senior Data Engineer to design, build, and maintain scalable data pipelines using Python & PySpark. You will optimize ETL/ELT workflows for structured and semi-structured data within our Azure ecosystem.

Key Responsibilities:

• Ingest, process, and store large datasets using Microsoft Azure services (ADF, ADLS, Synapse Analytics).
• Write efficient SQL queries for complex analysis, transformation, and reporting.
• Collaborate with data scientists and analysts to deliver reliable datasets.
• Implement robust data quality checks, monitoring, and performance tuning for batch and streaming workloads.
• Maintain best practices in data security, governance, and documentation.

Required Skills:

• Programming: Strong hands-on experience with Python; Solid expertise in PySpark/Apache Spark including transformations, joins, aggregations, and window functions.
• Databases & SQL: Advanced SQL (CTEs, indexing) and experience with relational/analytical databases.
• Azure Cloud: Proficiency with Azure Data Factory, ADLS, Synapse Analytics, Databricks, and cloud architecture principles.
• Concepts: Deep understanding of ETL vs ELT, data modeling (Star/Snowflake), partitioning, caching, and handling large-scale datasets.

To Have:

• Experience with streaming tools like Kafka or Event Hubs.
• Knowledge of CI/CD pipelines for data workflows.
• Familiarity with Airflow, scheduling in ADF, DevOps/Git practices, and basic data governance compliance.