Senior Data Engineer

About the Role

We are seeking a Senior Data Engineer to spearhead the development of our cutting-edge Data Lakehouse architecture. This pivotal role involves designing, building, and optimizing massive-scale, low-latency data pipelines essential for powering real-time analytics and advanced Machine Learning applications. Join our dynamic team in Kolkata and shape the future of data at Zorba Consulting India Pvt. Ltd.

Key Responsibilities

• Design and implement highly optimized, production-grade ETL/ELT pipelines utilizing Apache Spark (PySpark/Scala) for processing petabytes of data.
• Architect and manage a robust Data Lakehouse environment leveraging open-source technologies such as Delta Lake or Apache Hudi to ensure ACID transactions and maintain high data quality standards.
• Integrate and process real-time data streams efficiently using technologies like Apache Kafka or Kinesis.
• Develop and deploy automated data quality checks, comprehensive monitoring systems, and lineage tracking mechanisms across all data products.
• Collaborate closely with the infrastructure team to automate the deployment and scaling of our data platform on the cloud (AWS EMR/Glue or Databricks) using Infrastructure as Code principles with Terraform.
• Optimize data warehousing and querying performance within leading platforms such as Snowflake or Google BigQuery.

Technical Skills Required

• Expert proficiency and performance tuning experience with Apache Spark (PySpark or Scala).
• Mandatory hands-on experience with Data Lakehouse technologies (Delta Lake, Iceberg, or Hudi).
• Strong experience with at least one major public cloud data platform (AWS, GCP, or Azure).
• Solid understanding of data modeling techniques (Dimensional, Data Vault) and advanced SQL capabilities.
• Demonstrated experience with workflow orchestration tools like Apache Airflow or Prefect.