Senior Data Engineer

Mindera is seeking an experienced Senior Data Engineer to join their dynamic team in Chennai, Tamil Nadu, India. The ideal candidate will bring extensive expertise in big data technologies, ETL/ELT workflows, and data modeling techniques. This role focuses on designing and optimizing data pipelines, ensuring data integrity, and supporting analytics initiatives.

With 5 to 7+ years of relevant experience, you will be instrumental in developing and refining data pipelines, guaranteeing data fidelity, and advancing our analytics efforts.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark and Azure Databricks for data ingestion and processing.
• Implement and enhance real-time data streaming solutions.
• Optimize Spark job performance through effective memory management, partitioning, and efficient data storage.
• Collaborate with data scientists and analysts to understand data needs and deliver reliable datasets.
• Create and optimize complex SQL queries for data extraction, transformation, and analysis.
• Maintain data quality and integrity using automated testing and validation.
• Document data workflows and manage metadata for governance.
• Research and integrate new data engineering technologies to improve efficiency and scalability.

Mandatory Skills:

• PySpark for data processing and ETL.
• Azure Databricks for cluster management and platform usage.
• Data streaming concepts and frameworks like Spark Streaming.
• Strong Python programming skills for scripting and automation.
• Advanced SQL for database querying and management.
• Spark optimization techniques for performance enhancement.

Optional Skills:

• Snowflake for data warehousing.
• Cloud platforms such as AWS, Azure, or GCP.
• ETL/ELT principles, data modeling, and data warehousing.
• Big data tools including Kafka, Hadoop, and Hive.
• CI/CD practices and tools like Git and Jenkins.

Benefits:

• Flexible working hours.
• Competitive salary.
• Annual bonus based on company performance.
• Access to online training platforms like Udemy for continuous learning and professional growth.