Lead Pyspark Developer w Kraków, Polska

The Surveillance Enhancement Program (SEP) is a multi-year initiative aimed at developing and improving the surveillance platform to identify potential market misconduct, manipulation, and abuse. Our client fulfilled its commitment to the Financial Conduct Authority (FCA) by meeting Market Abuse Regulation (MAR) requirements for critical data sources in September 2017.

Subsequent projects have been launched to further enhance, expand, and complement the existing automated surveillance platform. This project will focus on alert enhancements, strategic data sourcing from FM order and trade lakes, and user interface and workflow improvements to comply with regulatory obligations such as Market Abuse Regulation and Dodd-Frank.

Additionally, the scope includes enhancements related to Control Room and Suitability Monitoring data, interfaces, and scenarios.

Responsibilities

Develop software applications based on business requirements

Maintain software applications and make enhancements according to project specifications

Participate in requirement analysis, design, development, testing, and implementation activities

Propose new techniques and technologies for software development.

Perform unit testing and user acceptance testing to evaluate application functionality

Ensure to complete the assigned development tasks within the deadlines

Work in compliance with coding standards and best practices

Provide assistance to Junior Developers when needed.

Perform code reviews and recommend improvements.

Review business requirements and recommend changes to develop reliable applications.

Develop coding documentation and other technical specifications for assigned projects.

Act as primary contact for development queries and concerns.

Analyze and resolve development issues accurately.

Skills

Must have

10+ years of experience in data intensive Pyspark development ,Window Functions , Performance Optimization ,Spark Architecture,Salting ,Column Pruning

Must have experience in leading a Development Team

Proven experience as a Core Python developer

Good understanding of Gen AI

Experience developing Classes, OOPS, exception handling, parallel processing

Strong knowledge of DB connectivity, data loading , transformation, calculation

Extensive experience in Pandas/Numpy dataframes, slicing, data wrangling, aggregations

Lambda Functions, Decorators

Vector operations on Pandas dataframes /series

Application of apply map, apply, map functions

Concurrency and error handling data pipeline batch of size [1-10 gb]

Ability to understand business requirements and translate them into technical requirements

Ability to design architecture of data pipeline for concurrent data processing

Working with Git source control Service-orientated architecture, including the ability to consider integrations with other applications and services

Debugging application

Nice to have

Knowledge of web backend technology

Django, Python, PostgreSQL.

Apache Airflow

Other

Languages

English: C1 Advanced

Seniority

Lead