Data Engineer

About the Job

A career at IBM Consulting involves long-term client relationships and global collaboration. You'll work with leading companies to shape their hybrid cloud and AI journeys using robust IBM technology, Red Hat, and strategic partners.

Your Role And Responsibilities

• Design Data Pipelines: Develop batch and real-time pipelines for Data Warehouse and Datalake using Google services like DataProc, DataFlow, PubSub, BigQuery, and Cloud Spanner.
• Develop Data Engineering Solutions: Utilize GCS, BigTable, Spark/Hadoop via DataProc, and Python/Beam via DataFlow to build solutions on the Google Cloud ecosystem.
• Manage Data Platforms: Schedule operations using Cloud Scheduler and Cloud Composer (Airflow).
• Implement Data Migration: Ensure seamless data transfer between systems using Google services.
• Optimize Data Layer: Design efficient storage and retrieval strategies for BigQuery, Big Table, and Cloud Spanner.

Required Expertise

• Google Cloud Ecosystem: Proficiency in DataProc, DataFlow, PubSub, BigQuery, etc.
• Data Pipeline Development: Experience with Airflow, dbt, Spark/Python or Scala for batch and real-time pipelines.
• Data Platform Management: Scheduling using Cloud Composer (Airflow).