Big Data Engineer - Python/Apache Spark

Draup is seeking a talented Big Data Engineer to join our dynamic team in Bengaluru/Bangalore, India. In this role, you will be instrumental in designing, building, and maintaining scalable data platforms and pipelines. You will collaborate with a highly skilled and enthusiastic group of developers, engineers, and product specialists, with the opportunity to significantly influence our data products.

What You Will Do

• Construct robust and scalable architectures for data storage, transformation, and analysis.
• Develop reliable, efficient, testable, and maintainable data engineering projects and pipelines.
• Design and implement scalable, generic, and reusable data solutions.
• Execute data warehousing, data mining, and data modeling tasks using agile methodologies.
• Lead and take full ownership of big data projects from inception through to production.
• Apply a problem-solving approach focused on understanding fundamental principles and implementing high-impact use cases.
• Collaborate effectively with data science, backend, data harvesting, and product teams.

What You'll Need

• A strong grasp of big data and distributed systems principles.
• Significant programming experience in Python.
• Mandatory proficiency in Apache Spark (PySpark).
• Solid experience in developing scalable ETL and ELT solutions from diverse data sources.
• Familiarity with relational and NoSQL datastores.
• Working and conceptual knowledge of MapReduce, HDFS, and Amazon S3.
• Ability to code and think within a functional programming paradigm.
• A passion for optimizing code performance and system resource utilization.
• Excellent communication skills, capable of explaining complex technical concepts to both technical and non-technical stakeholders.
• Demonstrated ability to take ownership of all technical aspects of software development for assigned projects.

What Will Give You An Advantage

• Expertise in big data infrastructure, distributed systems, data modeling, query processing, and relational databases.
• Experience designing big data solutions and working with various file storage formats such as Parquet, ORC, Avro, and Sequence files.
• Experience with and understanding of cluster managers like YARN, Spark Standalone, Mesos, or Kubernetes.
• Strong knowledge of data structures and algorithms.
• Ability to apply technologies to solve complex big data challenges and develop innovative solutions.
• A proactive problem-solving mindset with strong design and architectural pattern skills is preferred.
• Experience with AWS services such as EMR, Lambda, Glue, or equivalent services on other cloud platforms.
• Knowledge of workflow orchestration tools like Airflow or Jenkins.

Who You Are

• Educational background with a B.E / B.Tech / M.E / M.Tech / M.S in Computer Science or Software Engineering.
• 2-4 years of experience working with Big Data technologies.
• Enthusiastic about tackling the challenges of handling terabytes and petabytes of data daily.