AWS Data Engineer
Responsibilities
Qualifications & Requirements
Experience Level: Mid Level
Full Job Description
As an AWS Data Engineer - Associate Consultant at KPMG India, you will contribute to diverse projects within a collaborative and fast-paced environment, focusing on software development activities for KPMG India. You will be involved in the full software development lifecycle, including coding and unit testing. Collaborating closely with Technical Architects, Business Analysts, User Interaction Designers, and fellow Software Engineers, you will develop new product offerings and enhance existing ones, ensuring all development adheres to KPMG's best practices and procedures. This role demands a proactive approach to quickly learn and adapt to new technologies.
Responsibilities
Role: AWS Data Engineer
Location: Gurugram
Experience: 2 to 4 years
Key Responsibilities
- Databricks & Spark Development: Design, develop, and optimize ETL/ELT pipelines using Databricks (PySpark/Spark SQL). Build scalable data transformation workflows for both batch and streaming applications. Implement best practices for notebook development, orchestration, and job workflows.
- AWS Services Integration: Work extensively with AWS services including S3 (data lake storage), Lambda, Glue, Athena, EC2, IAM, CloudWatch, and optionally Step Functions. Deploy, monitor, and troubleshoot data workflows on the AWS platform.
- Data Integration and Quality: Integrate data from various sources such as databases, APIs, and flat files. Ensure high data quality through robust validation, logging, and error-handling frameworks within pipelines.
- Data Management: Work with REST APIs, relational databases (MySQL/PostgreSQL), and NoSQL stores like DynamoDB. Perform data modeling for data lakes, delta lakes, and analytical datasets. Optimize Spark jobs for enhanced performance and cost efficiency.
- Collaboration and Best Practices: Collaborate effectively with data analysts, data scientists, and business stakeholders. Maintain clear documentation, utilize version control systems (Git), and adhere to CI/CD practices.
- AWS Glue and CI/CD: Gain experience with AWS Glue ETL, Glue Catalog, or Glue Jobs. Exposure to CI/CD tools such as GitHub Actions, Bitbucket Pipelines, Azure DevOps, or Jenkins is valuable.
- DevOps and Streaming: Understand DevOps concepts and infrastructure-as-code tools like Terraform or CloudFormation. Knowledge of streaming technologies like Kafka or Kinesis is a plus.
Qualifications
Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
Company
KPMG India
KPMG entities in India are leading professional services firms affiliated with KPMG International Limited. Established in August 1993, KPMG India leverages a global network of firms to deliver service...