Lead Data Engineer

Ai Palette is seeking a Lead Data Engineer to join our engineering hub in Bengaluru, Karnataka, India. In this role, you will lead a team of data engineers, specializing in data engineering, providing technical guidance, mentoring, and performance feedback. You will be instrumental in optimizing and automating processes for our expansive consumer data lake, tailored for the CPG industry, driving higher automation and faster turnaround times for data refresh cycles (weekly, monthly, and quarterly). Your responsibilities will include enhancing data quality and completeness on the platform, designing, developing, and maintaining robust Spark data pipelines for efficient data acquisition, and establishing data engineering best practices and standards. You will also tackle data engineering challenges, including complex data structures, bottlenecks, and scalability issues. Collaboration with cross-functional teams such as data scientists, analysts, and software engineers will be key to understanding data requirements and developing scalable solutions. You will manage data extraction, transformation, and loading (ETL) tasks, monitor data pipeline performance, and ensure the reliability and scalability of our data infrastructure. Staying abreast of the latest advancements in data engineering is essential.

We are looking for individuals with a Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field, and 8+ years of proven experience as a Data Engineer. Strong programming skills in Python, Java, and Spark Pipelines, along with proficiency in SQL and various database technologies (PostgreSQL, MySQL, Oracle), are required. Familiarity with cloud platforms like AWS, Azure, or Google Cloud and their data services is essential. Experience with data modeling, data warehousing, ETL processes, and data warehouses such as Snowflake, Redshift, or Elastic Search is also necessary. You should possess strong analytical and problem-solving skills, excellent leadership and team management abilities, and effective communication skills to collaborate with both technical and non-technical stakeholders.

We offer great progression opportunities, health insurance, and sponsored training on MOOCs like Coursera and Udemy. Ai Palette is an equal opportunity employer, valuing diversity and ensuring a non-discriminatory workplace.