Senior Full Stack Developer

Job Purpose

Intercontinental Exchange, Inc. (ICE) is seeking an experienced Senior Data Engineer to join the AI Centre of Excellence (AI CoE) team in Hyderabad, Telangana. The AI CoE drives the adoption of artificial intelligence and advanced analytics across ICE's global exchange and financial data business, bringing together data engineering, machine learning, and applied research to deliver intelligent solutions at scale.

Responsibilities

• Own Apache Airflow end-to-end including DAG development for complex multi-step workflows, idempotency, backfill safety, SLA alerting, dynamic task mapping, and thorough testing. Review other engineers' work to raise team standards.
• Maintain Airflow on Kubernetes using KubernetesExecutor or CeleryKubernetesExecutor, managing Helm upgrades, pod templates, resource limits, and scheduler performance optimization.
• Build robust ETL and ELT pipelines from diverse sources (APIs, message queues, databases) that are fault-tolerant, incremental where applicable, and easy to debug.
• Treat data quality as a core responsibility by implementing schema validation, row-count reconciliation, and anomaly detection within pipelines.
• Design and maintain lakehouse architectures using Apache Iceberg (Bronze/Silver/Gold structure), managing partitioning strategies, time-travel queries, and schema evolution.
• Leverage Databricks for cluster management, Delta Lake workflows, Unity Catalog, and scheduled jobs to ensure cost-efficiency and organization.
• Develop dbt projects integrating with Airflow orchestration, creating well-structured models, tests, snapshots, macros, documentation, and freshness checks.
• Create reusable components such as pipeline templates, custom operators, and utility libraries for the shared platform.
• Partner with data scientists and ML engineers to deliver feature pipelines ensuring model training requirements are met regarding freshness and reproducibility.
• Instrument workloads with metrics, dashboards, alerts, runbooks, and participate in on-call coverage.
• Contribute actively to architecture discussions, tooling evaluations, and mentoring initiatives.