EY - GDS Consulting - AI and DATA -...

EY's Global Delivery Services (GDS) Consulting practice in Gurgaon, India, is seeking a skilled AWS Data Engineer - Senior to join our AI and Data team. If you're passionate about building a career that's as unique as you are, with the global scale, support, and inclusive culture to help you become your best, EY is the place for you. We value your unique voice and perspective in helping us build an exceptional experience for yourself and a better working world for all.

As an AWS Data Engineer - Senior, you will be instrumental in designing, developing, and optimizing robust data pipelines and solutions that power business intelligence, advanced analytics, and large-scale data processing. You will collaborate closely with data scientists, analysts, and other engineering teams to ensure seamless and efficient data flow across our systems, leveraging the power of the AWS cloud.

Key Responsibilities include:

• Designing and developing scalable ETL (Extract, Transform, Load) processes using AWS Glue to ingest and transform data from diverse sources into AWS Redshift or other storage solutions.
• Engineering governed, batch, and near real-time data pipelines using core AWS native technologies such as DirectConnect, S3, Lambda functions, Glue, Kinesis, and CloudTrail.
• Implementing serverless data engineering workloads within the AWS ecosystem, processing data from S3, RDS, and other cloud sources, applying business transformations using distributed compute services (e.g., EMR, Glue, Spark), and persisting insights in target stores like S3, Redshift, or DynamoDB.
• Maintaining, optimizing, and scaling AWS Redshift clusters to ensure high-performance data storage, retrieval, and query execution.
• Utilizing Amazon S3 for secure, scalable, and cost-effective data storage, managing large datasets, and integrating with other AWS services.
• Creating and managing AWS Glue crawlers and jobs for automated data cataloging and ingestion from various structured and unstructured data sources.
• Writing efficient Python and PySpark scripts for data transformation, integration, and automation, ensuring clean, reusable code.
• Implementing data validation, cleansing, and error-handling mechanisms within ETL pipelines to guarantee data accuracy and integrity.
• Optimizing AWS Glue jobs, Redshift queries, and data flows for maximum performance, reduced processing times, and cost efficiency.
• Enabling data consumption for reporting and analytics applications through AWS services like QuickSight, Sagemaker, and JDBC/ODBC connectivity.
• Designing logical data models, defining entities, relationships, constraints, and dependencies to support reporting and analytics use cases.
• Collaborating with data scientists, analysts, and stakeholders to understand data requirements and deliver solutions that drive data-informed decision-making.
• Developing and implementing monitoring strategies for data pipelines and proactively troubleshooting and resolving data-related issues.

Required Skills and Qualifications:

• 3-8 years of experience in data engineering or a related field, with a strong focus on AWS technologies.
• Academic background in Computer Science or a related discipline.
• Proficiency in PySpark, SQL, Stored Procedures, and Python programming.
• Extensive experience with AWS Glue for building ETL pipelines, managing crawlers, and working with the Glue data catalog.
• Strong experience with AWS Redshift, including cluster design, management, complex SQL query writing, and performance optimization.
• Hands-on experience with Amazon S3 for data storage, lifecycle policies, and integration with other AWS services.
• Solid Python programming skills, particularly for data manipulation (e.g., using pandas) and ETL job automation.
• Experience using PySpark within AWS Glue for large-scale data transformations.
• Proficiency in writing and optimizing SQL queries for data manipulation and reporting.
• Understanding of data warehouse concepts, including star schemas, partitioning, indexing, and data normalization.
• Strong analytical and problem-solving skills with meticulous attention to detail.
• Experience with version control systems like SVN or Git.
• Experience with data streaming technologies such as AWS Kinesis and Kafka implementation on AWS.

Good to have:

• Knowledge of AWS Identity and Access Management (IAM) for secure data resource access.
• Familiarity with DevOps practices and automation tools like Terraform or CloudFormation.
• Experience with data visualization tools like QuickSight or integrating Redshift data with BI tools (e.g., Tableau, PowerBI).
• Relevant AWS certifications such as AWS Certified Data Analytics – Specialty or AWS Certified Solutions Architect are advantageous.