Navtech

We are seeking a motivated Data Engineer with expertise in building scalable data lake architectures and high-performance data pipelines. Responsibilities include designing, building, and maintaining ETL/ELT pipelines using Apache Spark/PySpark, developing large-scale distributed data processing workflows, and managing modern data lake architectures with a Bronze/Silver/Gold layering approach. The role involves implementing Apache Iceberg features such as schema evolution, upserts, and time-travel queries, optimizing Spark jobs for performance, and managing Parquet file optimization. You will also design and maintain OpenSearch indexing strategies and cluster architecture, write and optimize advanced SQL queries, and implement orchestration workflows using modern scheduling tools. Ensuring data governance, security, and compliance through AWS IAM roles and best practices is crucial. Automation of workflows using Python scripting and CI/CD pipelines, and collaboration with Data Scientists, Analysts, and Product teams are key aspects of this role. Monitoring and troubleshooting production pipelines and data systems will also be required.

Who we are looking for:

• 1.5 - 4 years of experience in Data Engineering or Big Data Engineering.
• Strong hands-on experience with Apache Spark and PySpark.
• Proficiency with AWS services including S3, Glue, EMR, Athena, Lambda, EC2, and OpenSearch.
• Hands-on experience with Apache Iceberg (schema evolution, upserts, time travel).
• Solid understanding of Data Lake architecture and layered design (Bronze/Silver/Gold).
• Advanced SQL skills and query optimization techniques.
• Experience in Spark performance tuning.
• Strong understanding of Parquet file optimization and small-file handling.
• Experience with OpenSearch indexing and cluster design.
• Proficient Python programming for data processing and automation.
• Experience with ETL/ELT pipeline orchestration tools.
• Knowledge of AWS IAM roles, security practices, and data governance.
• Experience with CI/CD pipelines and managed AWS services.
• Hands-on experience handling large-scale distributed datasets.
• Excellent logical, analytical, and communication skills.
• Masters/Bachelors degree in Computer Science, Information Technology, Data Science, or related fields, with education conducted in English.