Data Engineer

Data Engineer at Iris Software - Noida

We are seeking a skilled Data Engineer with solid experience in data engineering and BI/software development, including ELT/ETL processes, data extraction, and manipulation within Data Lake/Data Warehouse environments. The ideal candidate will have proven experience in data modeling, analysis, and profiling, and has worked across diverse data domains. You will be responsible for continuous improvement and performance optimization to ensure high levels of data integrity, availability, and performance. Consulting and mentoring experience is a plus, as is experience working within an Agile environment and applying DevOps practices, SDLC, and CI/CD principles to build automated pipelines and deployments.

Key Responsibilities:

• Develop and maintain robust data pipelines using ELT/ETL processes.
• Perform data extraction, transformation, and loading into data warehouses and data lakes.
• Conduct data modeling, analysis, and profiling to ensure data quality and usability.
• Optimize data systems for performance, availability, and integrity.
• Collaborate with cross-functional teams and mentor junior engineers.
• Work in an Agile development environment.
• Implement DevOps practices for automated CI/CD pipelines.

Technical Skills:

• Proficiency in SQL and its Big Data variants (e.g., Hive-QL, Databricks).
• Strong programming skills in Python and Spark (PySpark, Spark SQL) or Scala.
• Experience with Bash/Shell scripting.
• Familiarity with source code control systems like GitHub or VSTS.
• Knowledge of Big Data technologies such as the Hadoop stack (HDFS, Hive, Impala, Spark).
• Experience with cloud Big Data warehouses including RedShift and Snowflake.
• Proficiency with UNIX command-line tools.
• Exposure to AWS services like EMR, Glue, Athena, Data Pipeline, and Lambda.

Mandatory Competencies:

• Python
• SQL
• Effective Communication
• PySpark
• CI/CD principles