Senior Data Engineer

Key Responsibilities

1. Data Platform Architecture & Engineering

• Design and maintain scalable data platforms for analytics and AI workloads.
• Build Lakehouse architectures using Apache Iceberg, Delta Lake, and Medallion Architecture.
• Develop data platform services and frameworks using Python (Flask, FastAPI), applying Object-Oriented Programming (OOP) principles and strong problem-solving skills.
• Implement data modeling, schema evolution, governance, and platform best practices.
• Ensure platform scalability, reliability, security, and performance.

2. Data Engineering & Processing

• Design, build, and optimize scalable ELT/ETL, batch, and real-time data pipelines using Python, dbt, Databricks, Snowflake, AWS Glue, GCP Composer, Dataflow, Dataproc, Apache Beam, Kafka, and Flink.
• Develop distributed data processing solutions for large-scale datasets, ensuring high performance, reliability, and throughput.
• Implement data quality, lineage, monitoring, validation, and governance processes across the data ecosystem.

3. Workflow Orchestration

• Develop and manage workflows using Airflow and/or Dagster.
• Implement CI/CD pipelines and DataOps best practices.
• Manage infrastructure using Terraform, Docker, and Kubernetes.
• Establish monitoring, alerting, and observability for production systems.

4. Collaboration & Ownership

• Collaborate with Data, Software, AI/ML, and Product teams.
• Own technical initiatives from design through deployment and maintenance.
• Mentor team members and promote engineering best practices.

5. AI & Machine Learning Data Infrastructure

• Build data pipelines for AI, ML, and RAG applications.
• Develop embedding, vector database, and feature store workflows.
• Support scalable data infrastructure for model training and inference.

Qualifications

• 4+ years of experience in Data Engineering required for this senior-level position.
• B.E./B.Tech/B.S. degree with significant prior experience, or equivalent hands-on expertise.
• Strong proficiency in Python (Flask, FastAPI), Object-Oriented Programming (OOP), and problem-solving skills essential for Ahmedabad/Pune tech environments.
• Proven experience building and managing large-scale ELT/ETL pipelines using dbt and Airflow/Dagster is mandatory.
• Hands-on experience with at least one cloud data platform (e.g., Snowflake, Databricks) and distributed data processing frameworks such as Apache Beam required.
• Good to have: Experience with Lakehouse architectures including Apache Iceberg, Delta Lake, and Medallion Architecture preferred for Gujarat market leaders.
• Familiarity with real-time data streaming technologies such as Kafka and Apache Flink is a strong advantage.
• Experience building data infrastructure for AI/ML and Retrieval-Augmented Generation (RAG) applications highly valued in current tech landscape.
• Strong understanding of vector databases, embedding pipelines, and feature stores beneficial for advanced roles.
• Proficiency with Terraform, Docker, Kubernetes, and CI/CD implementation expected for infrastructure ownership.
• Deep knowledge of data quality, observability, monitoring, and operational best practices necessary for enterprise scale.
• Excellent communication skills required to work independently, mentor team members, and drive end-to-end project execution with minimal supervision in Ahmedabad/Pune offices.