Octro

Data Engineer Opportunity at Octro Inc. in Noida, India

Octro Inc. is seeking a talented Data Engineer to design and maintain scalable data pipelines and analytics systems. This role is based in Noida, India, and offers a permanent position for an experienced professional.

Key Responsibilities:

• Design and implement robust, scalable data pipelines and analytics systems.
• Optimize data processing applications using Apache Spark, Python, and Scala.
• Develop and manage near real-time and streaming data processing with Apache Spark Structured Streaming.
• Integrate Apache Kafka for reliable real-time data ingestion and event-driven pipelines.
• Work with analytical and distributed data stores including ClickHouse, Trino/Presto, and data lake technologies like Delta Lake.
• Apply strong data modeling and metric design principles for large-scale analytics, including fact/dimension modeling and event-based schemas.
• Build and optimize ETL/ELT pipelines for data ingestion, transformation, aggregation, and performance.
• Write efficient, scalable, and maintainable code for large-scale data processing.
• Operate within on-premise or hybrid data platforms, focusing on cluster resource management, performance tuning, and capacity planning.

Required Skills and Experience:

• 2-4 years of experience in data engineering.
• Proficiency in Python and/or Scala with significant Apache Spark experience.
• Extensive experience with Apache Spark Structured Streaming.
• Strong hands-on experience with Apache Kafka.
• Experience with ClickHouse, Trino/Presto, and data lake technologies (Delta Lake or equivalent).
• Solid understanding of data modeling and metric design.
• Proven ability to design and implement ETL/ELT pipelines using Spark.
• Experience writing efficient, scalable code for large-scale data processing.
• Familiarity with on-prem or hybrid data platforms, cluster resource management, and performance tuning.

Preferred Skills:

• Experience with Elasticsearch for search, observability, or analytical use cases.
• Bachelor's degree in Computer Science, Software Engineering, or a related field, or equivalent practical experience.
• Strong familiarity with Git and collaborative development workflows.
• Working knowledge of cloud platforms (AWS, Azure, Google Cloud) for data services.
• Understanding of distributed data systems and database administration principles for analytical or NoSQL databases.