Senior Data Engineer

About the Role

We are seeking a talented Senior Data Engineer to spearhead the development of our innovative, next-generation Data Lakehouse architecture. This pivotal role involves the comprehensive design, construction, and optimization of massive-scale, low-latency data pipelines tailored for real-time analytics and sophisticated Machine Learning applications. Join us in Ahmedabad to shape the future of data at Zorba Consulting India Pvt. Ltd.

Key Responsibilities:

• Design and build highly optimized, production-grade ETL/ELT pipelines utilizing Apache Spark (PySpark/Scala) to process petabytes of data efficiently.
• Architect and manage our Data Lakehouse, leveraging open-source technologies like Delta Lake or Apache Hudi to ensure ACID transactions and maintain robust data quality.
• Integrate and process real-time data streams using cutting-edge technologies such as Apache Kafka or Kinesis.
• Implement automated data quality checks, comprehensive monitoring, and lineage tracking across all data products to ensure reliability and transparency.
• Collaborate closely with the infrastructure team to automate data platform deployment and scaling on cloud environments (AWS EMR/Glue or Databricks) using Infrastructure as Code principles with Terraform.
• Optimize data warehousing and querying performance within leading platforms like Snowflake or Google BigQuery to deliver fast and accurate insights.

Technical Skills Required:

• Expert proficiency and proven tuning experience with Apache Spark (PySpark or Scala).
• Mandatory hands-on experience with Data Lakehouse technologies (Delta Lake, Iceberg, or Hudi).
• Strong experience with at least one major public cloud data platform (AWS, GCP, or Azure).
• Solid knowledge of data modeling techniques (Dimensional, Data Vault) and advanced SQL querying capabilities.
• Experience with workflow orchestration tools such as Apache Airflow or Prefect is essential for managing complex data workflows.