Senior Pyspark Developer [Strong Py...

About the Role

Join our multi-year program (MYP) to build and enhance a surveillance platform designed to detect market misconduct, manipulation, and abuse under FCA/MAR regulations. You will focus on alerts enhancements, strategic data sourcing from FM order/trade lakes, UI/workflow improvements for regulatory compliance (MAR, Dodd Frank), Control Room & Suitability Monitoring, and scenario enhancements.

Key Responsibilities

• Develop and maintain software applications based on business requirements within project specifications.
• Participate in the full SDLC: requirement analysis, design, development, testing (unit/acceptance), and implementation.
• Propose new techniques and technologies for advanced data processing.
• Perform code reviews, ensure compliance with coding standards, and provide technical documentation.
• Analyze issues accurately and act as the primary contact for development queries.

Mandatory Skills

We are looking for a Pyspark Expert (10+ years experience) with deep knowledge of:

• Spark Architecture, Window Functions, Performance Optimization, Salting, Column Pruning.
• Core Python: OOPs, Classes, Exception Handling, Lambda/Decorators, Parallel Processing.
• Data Wrangling: Pandas/Numpy (Dataframes, Vector operations), DB connectivity, transformations.
• Pipeline Design: Concurrent processing for batches of 1-10 GB using Apache Airflow concepts.