Data Engineer for AZURE cloud Platf...

We are seeking a results-oriented Data Engineer with over 2 years of experience in developing data pipelines within cloud environments. This role involves designing, building, and optimizing Azure-based data ingestion and transformation pipelines using PySpark and Spark SQL. The ideal candidate will collaborate with cross-functional teams to deliver robust, scalable, and high-quality data solutions.

Responsibilities

• Design, develop, and maintain efficient ETL/ELT pipelines utilizing PySpark and Spark SQL.
• Construct and manage data workflows within the Azure ecosystem.
• Implement hybrid data integration solutions connecting on-premise databases to Azure Databricks via tools like Azure Data Factory (ADF), HVR, or Fivetran, ensuring secure network configurations.
• Optimize Spark jobs for enhanced performance, scalability, and cost-effectiveness.
• Establish and uphold best practices for data quality, governance, and comprehensive documentation.
• Partner with data analysts, data scientists, and business stakeholders to gather and refine data requirements.
• Contribute to CI/CD processes, automation, and utilize version control systems such as Git.
• Conduct root cause analysis, troubleshoot issues, and ensure the consistent reliability of data pipelines.

Qualifications

Required

• Bachelor's degree in Computer Science, Engineering, or a related discipline.
• Minimum of 2 years of practical data engineering experience.
• Proficiency in PySpark, Spark SQL, and distributed data processing.
• Strong understanding of Azure cloud services, including Azure Data Factory (ADF), Azure Databricks, and Azure Data Lake Storage (ADLS).
• Experience with SQL, data modeling, and performance tuning techniques.
• Familiarity with Git, CI/CD principles, and Agile methodologies.

Preferred

• Experience with orchestration tools like Airflow or Azure Data Factory pipelines.
• Knowledge of real-time streaming technologies such as Kafka, Azure Event Hub, or HVR.
• Exposure to API integrations, data connectivity, and cloud-native architectural patterns.
• Familiarity with large-scale enterprise data environments.