Senior Data Engineer

We are seeking a highly skilled Senior Data Engineer with over 3 years of hands-on experience building scalable data pipelines using Apache Spark and Databricks on the Azure cloud. This role is ideal for professionals proficient in PySpark, Spark optimization, Delta Lake, and modern data engineering practices who thrive working with large-scale datasets.

Key Responsibilities:

• Design, develop, and maintain high-performance batch and streaming pipelines using Databricks, Apache Spark (PySpark/SQL), Python, SQL, and Delta Lake within the Azure ecosystem.
• Leverage Azure services including Blob Storage, Data Factory, and Event Hubs to build robust data solutions.
• Implement Medallion Architecture patterns (Bronze/Silver/Gold) for scalable ETL/ELT frameworks.
• Create incremental processing pipelines and Change Data Capture (CDC) workflows optimized for TB+ datasets.
• Tune Spark workloads using advanced techniques such as partitioning, broadcast joins, caching, AQE, and data skew handling.
• Manage Delta Lake solutions ensuring ACID compliance, schema evolution, and pipeline reliability.
• Develop production-grade orchestration workflows using Databricks Jobs and Workflows with strong data quality controls.
• Enforce governance, security, and lineage standards via Unity Catalog and RBAC frameworks.
• Establish best practices for testing, monitoring, maintainability, and reusability across the data platform.
• Integrate platforms seamlessly with Power BI, Tableau, APIs, and custom applications to support reporting needs.
• Implement CI/CD pipelines using Git, Terraform, Azure DevOps, GitHub Actions, Jenkins, and Datadog for automated testing and alerting.
• Collaborate cross-functionally with engineering, product, and business teams to deliver trusted data solutions that enable self-service analytics.

Required Qualifications:

• 3+ years of hands-on expertise in Apache Spark and Databricks environments.
• Expert-level proficiency in PySpark/Scala with deep understanding of DataFrames, SQL, Structured Streaming, and distributed processing principles.
• Demonstrated experience optimizing high-performance Spark jobs for large-scale data (TB+) workloads.
• Solid background in both batch and streaming architectures within the Azure cloud ecosystem.
• Proven implementation of Medallion Architecture patterns for layered data management.
• Practical knowledge of incremental loads, CDC strategies, and schema evolution techniques.
• Strong foundational skills in SQL alongside modern Python engineering practices.

Nice to Have Skills:

• Familiarity with Azure Data Factory integration for hybrid orchestration scenarios.
• Experience managing streaming pipelines via Event Hubs or Apache Airflow external tools.
• Knowledge of NoSQL database systems alongside relational databases like MS SQL, PostgreSQL, and MySQL.
• Understanding of data governance frameworks, compliance standards (RBAC), security protocols, and lineage tracking using Unity Catalog.

Your Mission:

• Build for Scale:Create resilient Spark pipelines capable of handling massive datasets reliably.
• Build for Reliability:Institute rigorous data quality checks, automated testing, monitoring, and failure-handling protocols.
• Build for Change:Navigate complex schema evolutions, incremental loads, CDC patterns, and dynamic data sources with agility.
• Build for Governance:Leverage Unity Catalog, RBAC, lineage tools, and security best practices to ensure discoverable, trustworthy enterprise data.
• Build for Reuse:Develop modular frameworks and reusable components that accelerate development across multiple products.
• Build for Production:Maintain strict adherence to CI/CD workflows, Infrastructure as Code principles, observability standards, and operational excellence.