Data Engineer

About the Role

We are seeking a skilled Data Engineer specializing in Data Platforms on AWS. In this role, you will be instrumental in advising on, developing, and maintaining robust data engineering solutions within the AWS Cloud ecosystem. You will be responsible for the end-to-end design, build, and operation of both batch and real-time data pipelines, leveraging a comprehensive suite of AWS services.

Key Responsibilities

• Design and Develop Data Pipelines: Architect, construct, and manage batch and real-time data pipelines using core AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis, ensuring seamless integration and efficient operation of data engineering solutions.
• Create Data Layers: Develop and maintain data layers on AWS RedShift, Aurora, and DynamoDB, and facilitate data migration processes using AWS DMS.
• Manage Data Services: Schedule and oversee data services on the AWS Platform to ensure optimal performance and reliability of data engineering solutions.
• Develop Batch and Real-time Pipelines: Build and implement batch and real-time data pipelines essential for Data Warehouse and Data Lake environments, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.
• Utilize Open Source Technologies: Leverage leading open-source technologies including Apache Airflow and dbt, coupled with Spark using Python or Scala on the AWS Platform, to enhance data engineering solutions.

Required Education

Bachelor's Degree

Preferred Education

Master's Degree

Required Technical and Professional Expertise

• AWS Toolset Exposure: Demonstrable experience with AWS services like AWS EMR, AWS Glue, Glue Catalog, and Kinesis for designing, building, and operating data pipelines.
• Data Pipeline Development: Proven ability in developing batch and real-time data pipelines for Data Warehouses and Data Lakes, using AWS Kinesis and Managed Streaming for Apache Kafka.
• Data Layer Creation: Experience with AWS RedShift, Aurora, and DynamoDB for creating data layers and performing data migrations with AWS DMS.
• Open Source Technologies: Familiarity with open-source tools such as Apache Airflow and dbt, and proficiency with Spark (Python or Scala) on the AWS Platform.
• Data Service Management: Experience in scheduling and managing data services on the AWS Platform to ensure efficient operations.

Preferred Technical and Professional Experience

• AWS Glue Databrew Proficiency: Experience utilizing AWS Glue Databrew for data preparation and quality tasks within data engineering solutions.
• Lambda Functions Knowledge: Exposure to using Python-based Lambda functions for data processing and transformation tasks.
• RedShift Spectrum Familiarity: Experience with RedShift Spectrum for data warehousing and analytics purposes.

Experience

6 - 8 years of experience