Data Engineer

About the Role

We are looking for an experienced Data Engineer to design, build, and optimize scalable data infrastructure and pipelines. The ideal candidate will have strong expertise in ETL/ELT processes, distributed data processing, and data lake and warehouse architectures. You will work closely with cross-functional teams at PhysicsWallah (PW) to build reliable, high-performance data solutions that support business intelligence, analytics, and data-driven decision-making for our educational platform.

Key Responsibilities:

• Design and implement scalable, efficient ETL/ELT pipelines to ingest, transform, and process data from multiple sources in Noida or Bangalore locations.
• Architect and maintain robust data lake and warehouse solutions aligned with business needs of a top Indian EdTech firm.
• Own the development and optimization of distributed data processing systems using Spark, AWS EMR, or similar technologies within our cloud environment.
• Collaborate with cross-functional teams (data science, analytics, product) to gather requirements and implement data-driven solutions for student growth metrics.
• Ensure high levels of data quality, security, and availability across systems supporting millions of users.
• Evaluate emerging technologies and tools for data processing and workflow orchestration in the Indian tech landscape.
• Build reusable components, libraries, and frameworks to enhance engineering efficiency and reliability.
• Drive performance tuning, cost optimization, and automation of data infrastructure on AWS (S3, Redshift, Glue).
• Mentor junior engineers, review code, and set standards for development practices within our growing team in Noida or Bangalore.

Required Skills & Qualifications:

• 1-3 years of professional experience building scalable backend systems with a focus on performance, essential for India's competitive EdTech sector.
• Strong hands-on experience writing efficient, production-grade code in Python or Scala suitable for high-volume data environments.
• Deep understanding of core data engineering concepts: advanced data modeling, modern warehousing strategies (Snowflake/Redshift), scalable lakes (S3/Delta Lake), streaming vs. batch processing patterns, and metadata management protocols.
• Solid experience with AWS ecosystem (EMR, Glue, Lambda) or equivalent cloud platforms like GCP/Azure common in Indian startups.
• Experience working with orchestration tools like Apache Airflow (preferred for large-scale jobs), Azkaban, or Luigi to manage complex workflows.
• Proven expertise managing big data frameworks such as Apache Spark and orchestrating Kubernetes clusters for fault-tolerant deployments.
• Proficient in advanced SQL queries across relational databases (Postgres, Redshift) and NoSQL stores (MongoDB).
• Ability to understand API-driven architecture and integrate seamlessly with backend services powering our learning management systems.
• Demonstrated strong problem-solving skills, a proactive attitude towards ownership of critical data pipelines, and commitment to continuous improvement in high-growth environments.