Data Engineer - Bengaluru, India

Endpoint Clinical is seeking a highly skilled Data Engineer with deep expertise in Databricks, ETL/ELT pipeline development, and a strong command of SQL and Python. The ideal candidate will be proficient in the Medallion architecture and capable of building robust frameworks for auditability, data quality, and automated testing. Strong data analysis skills and experience with BI/reporting tools are highly preferred for this role in Bengaluru.

Responsibilities

• Lead the design, development, and optimization of ETL/ELT pipelines using Databricks, Python, Spark, and Delta Lake.
• Architect scalable data solutions using the Medallion architecture (Bronze, Silver, Gold layers).
• Design and implement data models and transformations using SQL and Python.
• Build and maintain audit frameworks to ensure traceability, compliance, and data lineage.
• Develop data quality monitoring and automated testing frameworks for pipeline reliability.
• Perform data analysis to support operational data requests and user queries.
• Collaborate with clinical data teams to analyse IRT/RTSM datasets.
• Create and maintain dashboards and reports using BI tools (e.g., Superset, Power BI, Tableau, Qlik, or similar).
• Assist in managing CI/CD and automated code branching/deployment processes.
• Ensure compliance with GxP, CDISC, and other regulatory standards.
• Mentor junior engineers and promote engineering best practices.

Required Experience

• 8–10 years of experience in data engineering, including leadership or team lead responsibilities.
• Strong hands-on experience with Databricks, Apache Spark, and Delta Lake.
• Advanced proficiency in SQL and Python for data transformation and automation.
• Experience with ETL/ELT orchestration, job optimization, and performance tuning.
• Proven experience designing and implementing audit, data quality, and testing frameworks.
• Hands-on experience with IRT/RTSM clinical trial data systems.
• Strong data analysis skills and the ability to interpret complex datasets.
• Experience with BI/reporting tools such as Power BI, Tableau, or Qlik.
• Knowledge of clinical data standards (e.g., CDISC, SDTM, ADaM).
• Experience with cloud platforms (Azure, AWS, or GCP) and CI/CD pipelines.

Desirable Skills

• Experience in Pharma or Life Sciences, particularly in clinical trial operations.
• Familiarity with data governance and metadata management tools.
• Certifications in Databricks, cloud platforms, or clinical data standards.
• Exposure to machine learning workflows or data science collaboration.

Education

• Bachelor’s degree in computer science, Engineering, or an equivalent field, or appropriate experience will be considered.