Senior Data Engineer
Full Job Description
Quantiphi is seeking a Senior Data Engineer for their Mumbai, Bangalore, or Trivandrum location. This is a hands-on role focused on building and operating data pipelines and transformation layers for regulated healthcare data. You will be responsible for ingestion pipelines using Apache Flink and PySpark, the dbt transformation layer, FHIR serialization, and FHIR-Repository integration. The role requires strong proficiency in Python and SQL, experience with Google Cloud Platform, dbt, Apache Iceberg, and Apache Kafka. Familiarity with healthcare data standards like HL7v2 and FHIR is essential. The ideal candidate will have a Bachelor's or Master's degree in Computer Science or a related field, with 3+ years of data engineering experience. Experience with AI coding assistants is also a plus.
Key Responsibilities
- Build and maintain ingestion pipelines for HL7v2, FHIR, CCDA, and CSV data using Apache Flink and PySpark.
- Implement source-format parsers in Python and write comprehensive tests.
- Manage synchronous Informatica MDM calls and asynchronous MDM event consumption.
- Develop the end-to-end dbt transformation layer, including staging, intermediate, CDM, and data product models.
- Implement and maintain a shared dbt macro library.
- Build the FHIR serialization layer, including Iceberg tables and FHIR Bundles for Kafka.
- Develop and maintain FHIR-Repository integration components.
- Orchestrate dbt runs, batch jobs, and maintenance operations using Cloud Composer DAGs.
- Work within a spec-driven development framework, drafting and refining unit specifications.
- Implement and monitor data quality checks at various pipeline stages.
- Participate in code reviews, on-call rotations, and incident response for operational stability.
- Optimize pipeline performance, including Flink, Iceberg, and dbt configurations.
- Document all components built through the SDD framework.
Required Skills and Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
- 3+ years of hands-on data engineering experience.
- Strong proficiency in Python and SQL. PySpark and PyFlink familiarity are strongly preferred.
- Hands-on experience with Google Cloud Platform services like Cloud Storage, Dataproc, Cloud Composer, Cloud Run/GKE, Secret Manager, and IAM.
- Production experience with dbt, including incremental materialization, custom macros, tests, and project organization.
- Production experience with Apache Iceberg, including table creation, partitioning, compaction, and schema evolution.
- Experience with Apache Kafka, including producers, consumers, and integration with stream processors.
- Experience with streaming data processing, preferably Apache Flink.
- Familiarity with healthcare data standards such as HL7v2 and FHIR R4.
- Experience with version control (Git) and CI/CD pipelines.
- Comfort working with AI coding assistants.
- Strong problem-solving skills, debugging discipline, attention to detail, and ability to work in an agile team environment.
Nice-to-Have Skills
- FHIR-Repository or HAPI FHIR experience.
- Informatica MDM experience.
- Atlan experience.
- Java or Kotlin proficiency.
- Apache Flink production operational experience.
- Experience with FHIR profile validation tools.
- Experience with spec-driven or contract-first development workflows.
- Experience with Iceberg catalog alternatives like Polaris, Nessie, or Snowflake Open Catalog.
Company
Quantiphi
Quantiphi is a technology company that fosters a global and diverse culture built on transparency, integrity, learning, and growth. We are committed to providing an environment where individuals can i...