Data Engineer

About the Role

We are seeking an experienced Data Engineer (7+ years) with deep expertise in Azure and Databricks platforms to join our dynamic team. The ideal candidate will lead the design, development, and maintenance of scalable data pipelines while optimizing processing workflows for maximum reliability.

Key Responsibilities:

• Data Pipeline Development: Build robust ETL processes using Azure Data Factory and Databricks with a focus on automation and orchestration.
• Dynamic OpCo Deployments: Refactor code and RESTful services to support multitenant scalability across the platform for various operating companies.
• Collaboration & Stakeholder Management: Partner closely with data scientists, analysts, and business leaders to translate complex requirements into technical specifications that fuel analytics and AI initiatives.
• Optimization & Scalability: Enhance data processing workflows for performance; implement best practices for efficient storage and retrieval of large-scale structured, semi-structured, and unstructured datasets.
• Data Quality & Validation: Implement rigorous checks to ensure accuracy and integrity. Continuously refine testing strategies to maintain high standards across all pipelines.
• Monitoring & Troubleshooting: Proactively identify issues to ensure timely data delivery for business intelligence and machine learning models.
• HDP Integration: Execute new pipeline integrations with the HDP Upload Portal to streamline ingestion processes.
• Security & Compliance: Uphold industry standards by implementing governance frameworks, managing access controls, and ensuring full data compliance.