Senior Data Engineer

Join our Data Engineering team in Bengaluru, Karnataka, India, as a Senior Data Engineer. You will play a crucial role in designing, developing, and maintaining robust data solutions within on-premise environments. This position involves close collaboration with internal teams and client stakeholders to build and optimize data pipelines and analytical tools, utilizing technologies such as Python, Scala, SQL, Spark, and the Hadoop ecosystem. This role specifically demands deep hands-on experience with big data technologies in traditional data center environments, distinct from cloud-based solutions.

Responsibilities:

• Design, build, and maintain on-prem data pipelines for ingesting, processing, and transforming large datasets from various sources into data warehouses and data lakes.
• Develop and optimize Scala-Spark and SQL jobs for high-performance batch and real-time data processing.
• Ensure the scalability, reliability, and performance of data infrastructure within an on-prem setup.
• Collaborate with data scientists, analysts, and business teams to translate their data requirements into effective technical solutions.
• Troubleshoot and resolve issues within data pipelines and data processing workflows.
• Monitor, tune, and improve Hadoop clusters and data jobs for optimal cost and resource efficiency.
• Stay abreast of on-prem big data technology trends and propose enhancements to elevate data engineering capabilities.

Qualifications:

• Bachelor's degree in software engineering or a related field.
• A minimum of 5 years of experience in data engineering or a similar domain.
• Strong programming proficiency in Python and Scala.
• Expertise in SQL with a comprehensive understanding of data warehousing concepts.
• Hands-on experience with Hadoop ecosystem components, including HDFS, Hive, Apache Hudi, Iceberg, and Delta Lake.
• Demonstrated ability to design and manage data solutions in on-prem environments, with no reliance on cloud platforms.
• Experience with third-party data integrations from diverse sources, including APIs.
• Proficiency in using Airflow or comparable orchestration tools.
• Excellent problem-solving skills, with the capacity to work both independently and collaboratively.
• Strong communication skills, enabling effective engagement with technical and non-technical stakeholders.