Senior Data Engineer

Celebal is seeking a skilled Senior Data Engineer with excellent communication abilities and practical experience in modern data engineering tools and technologies. This role involves designing, developing, and optimizing scalable data pipelines to transform raw data into actionable business insights. The Senior Data Engineer will collaborate with cross-functional teams to deliver data-driven solutions.

Key Responsibilities:

• Interpret business requirements and coordinate with internal teams and stakeholders for successful project delivery.
• Build, maintain, and optimize scalable data pipelines using Databricks and PySpark.
• Develop robust data processing solutions for large and complex datasets.
• Write efficient SQL queries for data extraction, transformation, validation, and reporting.
• Design and implement ETL workflows using Python and PySpark.
• Perform data joins, aggregations, filtering, and cleansing to ensure high-quality data outputs.
• Monitor and troubleshoot pipeline performance to enhance efficiency and reliability.
• Ensure data accuracy, consistency, and adherence to governance standards across systems.
• Collaborate closely with analysts, developers, and business teams to meet data requirements.

Required Skills:

• Strong communication and stakeholder management skills.
• Hands-on experience with Databricks.
• Strong proficiency in PySpark.
• Advanced SQL skills for large-scale data processing.
• Experience in Python-based ETL development.
• Good understanding of data modeling and data warehousing concepts.
• Experience in optimizing queries and workflows for performance.

Preferred Qualifications:

• Experience with cloud platforms such as Azure or AWS.
• Familiarity with version control tools like Git.
• Knowledge of scheduling/orchestration tools such as Airflow or Azure Data Factory.
• Understanding of CI/CD practices in data engineering.

Experience:

• 2 - 7 years of relevant experience in Data Engineering.

Location:

• Pune, India

Immediate joiners are preferred.