Data Engineer

Responsibilities

• Design, develop, and maintain robust data pipelines using Airflow, DBT, and Python within a BigQuery environment.
• Implement and maintain data integrations with various sources and targets using tools like Airbyte, Elementary, and Metaflow, focusing on optimizing data storage and retrieval in BigQuery.
• Write advanced SQL queries and scripts for data manipulation and retrieval in BigQuery.
• Develop and optimize data models and schemas in Preset and other visualization tools tailored for performance in BigQuery.
• Utilize Pyspark for big data processing tasks, ensuring compatibility with BigQuery's infrastructure.
• Work with APIs, Docker, and GCP services to enhance and automate data workflows, particularly focusing on integration with BigQuery.
• Apply best practices in continuous integration and continuous deployment (CI/CD) environments using testing and pipeline validation techniques.
• Employ asynchronous architectures, queues, caching strategies, and distributed computing to enhance system performance.
• Write custom code in Python and utilize Jinja within DBT for dynamic pipeline generation specifically designed for efficiency in BigQuery.
• Adopt design patterns and engineering best practices to ensure code scalability and maintainability.

Key Outcomes

• Develop a Unified Data Foundation: Spearhead the creation of a single source of truth that centralizes data assets, ensuring consistency and reliability across the board.
• Drive Impact Across Teams: Your work will directly influence the success of business analysts, data scientists, and machine learning engineers by providing crucial data to excel in their roles.
• Optimize Business Logic: Design and manage data tables that support business logic functions, delivering tailored data solutions that drive strategic decision-making.