Senior Data Engineer

Findem is seeking an experienced Senior Data Engineer specializing in Big Data and Data Pipelines for its Delhi and Bengaluru offices. This role involves building, deploying, and managing diverse data pipelines, data lake, and big data processing solutions using cutting-edge big data and ETL technologies.

The ideal candidate will possess strong core engineering skills and be highly proficient in AI-assisted coding, utilizing tools like Cline or Cursor as part of a modern engineering workflow. Responsibilities include constructing robust data pipelines, big data processing solutions, and data lake infrastructure. You will assemble and process large, complex datasets from various sources such as MongoDB, S3, Server-to-Server, and Kafka, utilizing SQL and big data technologies. The role also involves building analytical tools and self-serve query tools for actionable business insights, developing data models and schemas for optimal performance and scalability, and managing data transformation, metadata, dependency, and workflow processes.

Key responsibilities include building data pipelines, big data processing solutions, and data lake infrastructure using various big data and ETL technologies. You will assemble and process large, complex data sets that meet functional and non-functional business requirements, ETL from a wide variety of sources like MongoDB, S3, Server-to-Server, Kafka etc., and processing using SQL and big data technologies. Additionally, you will build analytical tools to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics, as well as build interactive and ad-hoc query self-serve tools for analytics use cases. Building data models and data schemas for performance, scalability, and functional requirements, along with processes supporting data transformation, metadata, dependency, and workflow management, are also key duties. Research, experimentation, and prototyping of new tools and technologies are encouraged.

Required skills include strong proficiency in Python or Scala and AI-assisted coding. Experience with big data technologies like Spark, Hadoop, Athena/Presto, Redshift, and Kafka is essential, along with familiarity with file formats such as Parquet, JSON, Avro, and ORC. Experience with workflow management tools like Airflow and familiarity with batch processing, streaming, and message queues are required. Proficiency with visualization tools like Redash, Tableau, or Kibana, and experience with structured and unstructured datasets are also necessary, along with strong problem-solving skills.

Good to have skills include exposure to NoSQL databases like MongoDB, cloud platforms such as AWS or GCP, microservices architecture, and machine learning techniques.

This is a full-time, hybrid role requiring 3 days onsite in Delhi, India. Findem offers full benefits and is globally headquartered in the San Francisco Bay Area, with India headquarters in Bengaluru.