Senior Big Data Engineer – ClickHou...

About the Role

We are seeking an experienced Senior Big Data Engineer to design, build, and operate large-scale data pipelines that transform high-volume raw data into reliable, query-ready datasets. You will own end-to-end data infrastructure from ingestion through warehousing, orchestration, quality assurance, and observability.

Key Responsibilities

• Pipeline Orchestration: Design robust batch and streaming pipelines using Apache Airflow DAGs with advanced scheduling, backfills, idempotency, and failure handling strategies.
  Data Modeling & Transformation: Develop analytics-ready datasets using dbt following strict layered architectures (staging, intermediate, mart) with comprehensive testing and documentation.
• ClickHouse Expertise: Own the primary analytical store including schema design for MergeTree engines, partitioning strategies, materialized views, distributed/replicated tables, and query/memory optimization.
  Cloud & NoSQL Integration: Leverage BigQuery for warehouse patterns and manage key-value stores like DynamoDB, Redis, or Bigtable based on access requirements.
• Data Quality & Observability: Implement frameworks for validation, freshness monitoring, anomaly detection, and maintain structured logging/alerting systems.
  Engineering Excellence: Build containerized Docker environments, manage CI/CD pipelines, handle schema evolution safely, and ensure SLA adherence.

Requirements

• Experience: 4-6 years in data engineering with production-grade Python/SQL skills.
  Core Skills: Expert-level ClickHouse (MergeTree family), Apache Airflow, dbt, BigQuery, and NoSQL databases.
  Cloud & Tools: GCP/AWS experience, Docker, Git, CI/CD practices.
• Nice-to-Have: Experience with Elasticsearch/OpenSearch, Kafka/Pub/Sub, Spark/S3 data lakes, Terraform/Kubernetes, and Go programming.
  Soft Skills: Strong ownership mindset, problem-solving abilities, and effective cross-team collaboration skills.