Principal Engineer

Marsh Corporate is seeking a Principal Engineer specializing in Data Engineering for their Tech team. This hybrid role is based in Noida, Uttar Pradesh, requiring at least three days a week in the office. The position involves designing and implementing scalable data pipelines and AI-based solutions using Databricks, focusing on end-to-end ETL/ELT processes, managing large datasets, and optimizing data for analytical use with tools like Python, PySpark, and AWS S3.

Key responsibilities include developing and maintaining data pipelines with Databricks and the Medallion Architecture, designing AI solutions with Databricks Genie, and integrating Databricks features via API. You will write data transformation scripts in Python and PySpark, manage real-time data in AWS S3, and utilize SQL for data manipulation. Collaboration with cross-functional teams, monitoring pipeline performance, troubleshooting, and documenting processes are also crucial. The role requires ingesting and processing both batch and streaming data from structured and unstructured sources.

Required qualifications include experience with Databricks components (Pipeline, Genie, Unity Catalog, Datawarehouse), proficiency in Python, PySpark, and SQL, and experience with AWS S3 data. Familiarity with Data Governance, data access security, configuring Databricks Job compute, and version control using Git is essential. Understanding of Databricks API integration, bulk data and real-time data streaming, Delta Lake, and other Databricks technologies is expected. Knowledge of additional AWS services such as Athena, Glue, Lambda, DMS is beneficial.

The company offers a hybrid work environment, opportunities for professional development, and a chance to work with global stakeholders. Marsh fosters an inclusive culture and provides a range of career opportunities and well-being benefits.