Team Lead Data Engineer
Responsibilities
Qualifications & Requirements
Experience Level: Senior Level
Full Job Description
Mindera is seeking a highly skilled and proactive Data Engineer to join their team in Chennai, Tamil Nadu, India. The ideal candidate will possess strong hands-on expertise with AWS cloud services, Databricks, Apache Spark, Python, and SQL, coupled with excellent communication and stakeholder engagement skills. This role is crucial in designing, building, and optimizing data pipelines and analytics solutions to support critical business initiatives.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and transformation workflows using Databricks and Spark.
- Build and optimize ETL/ELT processes to ingest, cleanse, and prepare large datasets.
- Utilize AWS services such as S3, Lambda, IAM, ECS, and CloudWatch to support data architecture and operational needs.
- Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and translate them into technical solutions.
- Ensure data quality, reliability, and performance across all data systems.
- Implement and enforce best practices for coding, version control, CI/CD, and environment management.
- Monitor and troubleshoot data pipelines, ensuring high availability and timely delivery.
- Contribute to architectural decisions, documentation, and process improvements.
Requirements
- Strong hands-on experience with AWS Cloud (S3, Lambda, IAM, EC2 or equivalent).
- Extensive experience with Databricks, Delta Lake, and Unity Catalog.
- Proficiency in Apache Spark, with PySpark being preferred.
- Strong programming skills in Python.
- Excellent command of SQL, including performance tuning and working with large datasets.
- Experience working in fast-paced, agile environments.
- Strong problem-solving abilities and a proactive approach to identifying improvements.
- Excellent communication and stakeholder management skills; ability to collaborate across multiple teams.
- Experience with data governance frameworks and tools (e.g., DataHub, Soda).
- Experience with CI/CD tools (GitHub Actions).
Availability
Candidates must be available to support meetings and calls during US Pacific Time (PT).
Personal Attributes
- Proactive, self-driven, and highly accountable.
- Strong ownership mindset with the ability to work independently.
- Comfortable engaging with technical and non-technical stakeholders.
- Passion for building reliable, scalable, and high-quality data systems.
What We Offer
- A fun, happy, and politics-free work culture built on the principles of lean and self-organisation.
- Opportunity to work with large-scale systems powering global businesses.
- Competitive salary and benefits.
Company
Mindera
Mindera is a technology company specializing in building software engineering applications, including web and mobile. They partner with clients to deliver high-performance, resilient, and scalable sof...