Python Developer with Azure Databricks

You will join the team behind an internal AI platform for processing and interacting with unstructured data. The team is currently over 30 people strong and is organized into agile teams, each of which is self-sufficient and handles the creation of features from the idea stage, through analysis, implementation, testing, production deployment, and maintenance. The team is international, and it's located in Krakow, Wroclaw and London

Responsibilities

Maintain and enhance the existing Python-based reporting system, ensuring stability and accuracy in regulatory reporting.

Monitor, troubleshoot, and resolve issues in the current reporting workflows.

Ensure that reports comply with regulatory requirements and meet audit standards.

Lead the migration of the existing reporting system to Azure Databricks, ensuring a smooth transition.

Design and implement new, scalable reporting solutions using PySpark, SQL, and Databricks visualization tools.

Optimize data processing pipelines to enhance performance, reliability, and efficiency of regulatory reports.

Optimize PySpark performance to handle large-scale financial data processing for regulatory reporting.

Improve SQL query execution to ensure fast, efficient report generation.

Implement best practices for structured and unstructured data processing.

Automate reporting processes to reduce manual intervention and enhance accuracy.

Work with DevOps teams to ensure seamless Azure Databricks integration into the existing infrastructure.

Skills

Must have

Databricks Concepts

Databricks performance optimizations

ACID properties of RDMS

Python, PySpark

SQL Concepts

Databricks Queries and Visualization

Exposure to CI/CD DevOps Pipelines

Nice to have

Azure Data Services

ETL & Data Processing,

Financial Data Processing, experience handling risk, compliance, and regulatory data in an investment banking environment.

Data Modeling, meta data management

Git, GitHub, GitLab,

Jenkins

Regulatory compliance knowledge: Basel, MiFID, GDPR

Big Data

Cloud security and access controls (IAM, RBAC)

Familiarity with Docker, Kubernetes, Apache

Other

Languages

English: C1 Advanced

Seniority

Senior