Senior Python & Spark Developer

About the Role

We are seeking a seasoned Senior Python & Spark Developer with extensive experience in Python (Django, Flask) and PySpark for building and managing large-scale data processing systems. The ideal candidate is a proactive self-starter capable of independently managing and delivering projects from inception to completion, while also providing mentorship and guidance to junior team members.

Key Responsibilities

• Design, develop, and maintain robust, scalable data-driven and event-driven applications.
• Build backend services utilizing Python frameworks like Django and Flask, integrating them with high-performance distributed data processing pipelines powered by PySpark.
• Take full ownership of assigned tasks, managing the entire lifecycle from initial design through to final deployment.
• Collaborate effectively with cross-functional teams to define requirements, participate in design discussions, and contribute to delivery planning.
• Mentor and guide junior developers, instilling best practices in coding and problem-solving.
• Design and optimize data storage solutions using both Relational Database Management Systems (RDBMS) and NoSQL databases.
• Implement event-driven architectures by integrating applications with messaging services such as Kafka and MQ.
• Ensure seamless application deployments via Continuous Integration and Continuous Deployment (CI/CD) pipelines using Jenkins and Docker.

Core Skills & Technologies

• Languages: Python 3
• Frameworks: Apache Spark (PySpark), Django, Flask
• Databases: PostgreSQL (RDBMS), Cassandra, MongoDB (NoSQL)
• Messaging: Kafka, MQ
• Architecture: Event-Driven, Data-Driven
• CI/CD Tools: Jenkins, Docker
• Monitoring tools: ELK, Prometheus, Grafana

Good to Have

• Familiarity with Data Lake and Data Warehouse concepts.
• Experience with large-scale distributed systems.
• Exposure to cloud platforms such as GCP, AWS, or Azure.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field.
• A minimum of 5 years of professional software development experience.
• At least 2 years of hands-on experience with PySpark for big data processing.
• Significant backend development experience using Django and Flask.
• Demonstrated ability to work autonomously and assume end-to-end responsibility for tasks.
• Exceptional problem-solving, communication, and mentoring capabilities.