Data Engineer

Lifesight is seeking a skilled Data Engineer in Bangalore to join their growing Data and Business Intelligence organization. The role involves deep data engineering projects and offers the chance to influence technical strategy and team culture. Responsibilities include designing and building data platforms and services within cloud environments to support strategic business decisions across Lifesight's products.

A successful candidate will be a proactive individual, proficient in various big data technologies, able to collaborate with other engineers, and mentor team members.

Responsibilities:

• Design and implement scalable, fault-tolerant data processing systems (batch and streaming) processing hundreds of terabytes of data per day, including a petabyte-sized data warehouse and an Elasticsearch cluster.
• Develop high-quality data solutions and simplify diverse datasets.
• Build data pipelines to optimize data quality and handle poor-quality data sources.
• Take ownership of data mapping, business logic, transformations, and data quality.
• Conduct low-level systems debugging, performance measurement, and optimization on large production clusters.
• Participate in architectural discussions, influence the product roadmap, and take responsibility for new projects.
• Maintain and support existing platforms while adapting to new technology stacks and architectures.

Requirements:

• Proficiency in Python and Pyspark.
• Strong understanding of Apache Spark, Spark tuning, creating RDDs, and building data frames.
• Create Java/ Scala Spark jobs for data transformation and aggregation.
• Experience with big data technologies like HDFS, YARN, MapReduce, Hive, Kafka, Spark, Airflow, Presto, etc.
• Experience in building distributed environments using Kafka, Spark, Hive, Hadoop, etc.
• Good understanding of distributed database systems.
• Experience with various file formats like Parquet, Avro, etc, for large volumes of data.
• Experience with one or more NoSQL databases.
• Experience with AWS or GCP.
• 5 years of professional experience as a data or software engineer.

Lifesight empowers brands and enterprises to make data-driven decisions. Their Data Intelligence Platform helps companies in advertising, retail, CPG, QSR, and automotive industries optimize media spend, improve customer experiences, and drive growth. Located in Bangalore, Lifesight is seeking a Lead Product Manager to lead product vision and development for geospatial, consumer, mobility, and sales data intelligence solutions, offering a unique opportunity to shape how businesses leverage data for strategic advantage.