Data Engineer

About the Role

Location: Chennai (Preferred) / Pune / Hyderabad / Bangalore

• Contract Duration: 6 Months
• Shift: UK Shift (2:00 PM – 11:00 PM)

Overview

We are seeking a hands-on Data Engineer to develop, optimize, and maintain automated data pipelines supporting advanced governance and analytics initiatives. This role focuses on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration using Azure Databricks, Azure Data Lake, and MS Purview.

Key Responsibilities:

• Pipeline Development: Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance metadata.
• Data Quality Capture: Implement profiling logic to capture granular issue-level metadata for targeted remediation.
• Governance Metrics Automation: Develop pipelines generating metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query costs, retention policies, and sensitive data mapping.
• MS Purview Integration: Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for governance reporting.
• Data Retention & Policy Enforcement: Implement logic for tracking retention periods and policy compliance (masking, RLS).
• Job & Query Monitoring: Build pipelines to track job performance, SLA adherence, and query costs for optimization.
• Collaboration: Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.