Data Platform Engineer

About the Role

As a Data Platform Engineer at Accenture, you will play a crucial role in shaping the future of our data infrastructure. You will be instrumental in assisting with the data platform blueprint and design, encompassing all relevant data platform components. Your responsibilities will include collaborating closely with Integration Architects and Data Architects to ensure seamless and cohesive integration between various systems and data models. You will also engage in strategic discussions to refine and enhance the overall data architecture, ensuring it aligns with the organization's data needs and objectives throughout the data platform lifecycle.

Key Responsibilities:

• Develop high-quality, scalable ETL/ELT pipelines utilizing Databricks technologies such as Delta Lake, Auto Loader, and DLT.
• Demonstrate excellent programming and debugging skills in Python.
• Exhibit strong hands-on experience with PySpark for building efficient data transformation and validation logic.
• Be proficient in at least one major cloud platform: AWS, GCP, or Azure.
• Create modular dbx functions for transformation, PII masking, and reusable validation logic across DLT and notebook pipelines.
• Implement robust ingestion patterns using Auto Loader, incorporating checkpointing and schema evolution for structured and semi-structured data.
• Build secure and observable DLT pipelines featuring DLT Expectations, supporting a Bronze/Silver/Gold medallion layering approach.
• Configure Unity Catalog, including setting up catalogs, schemas, managing user/group access, enabling audit logging, and defining masking for PII fields.
• Ensure secure data access across domains and workspaces using Unity Catalog External Locations, Volumes, and lineage tracking.
• Leverage data assets from the Databricks Marketplace for enrichment, model training, or benchmarking purposes.
• Collaborate with data sharing stakeholders to implement Delta Sharing, both internally and externally.
• Integrate business intelligence tools like Power BI, Tableau, and Looker with Databricks using optimized connectors (ODBC/JDBC) and Unity Catalog security controls.
• Build stakeholder-facing SQL Dashboards within Databricks to monitor key performance indicators (KPIs), data pipeline health, and operational service level agreements (SLAs).
• Prepare GenAI-compatible datasets, manage vector embeddings, index with Databricks Vector Search, and utilize Feature Store with MLflow.
• Package and deploy pipelines using Databricks Asset Bundles through CI/CD pipelines in GitHub or GitLab.
• Troubleshoot, tune, and optimize jobs using the Photon engine and serverless compute, focusing on cost efficiency and SLA reliability.
• Gain experience with cloud-based services pertinent to data engineering, data storage, data processing, data warehousing, real-time streaming, and serverless computing.
• Apply performance optimization techniques effectively.
• Understand and apply data modeling and data warehousing principles.

Good to Have:

• Databricks Certified Professional or similar certifications.
• Knowledge of machine learning concepts and experience with popular ML libraries.
• Familiarity with big data processing frameworks (e.g., Spark, Hadoop, Hive, Kafka).
• Experience with data orchestration tools like Apache Airflow.
• Understanding of CI/CD pipelines and DevOps practices in a cloud environment.
• Experience with ETL tools such as Informatica, Talend, Matillion, or Fivetran.
• Familiarity with dbt (Data Build Tool).

Required Skills:

• Databricks Unified Data Analytics Platform

Experience Required:

• A minimum of 5 years of relevant experience is required.

Educational Qualification:

• 15 years of full-time education is required.