Senior Data Engineer (Azure, Databricks) w Wrocław, Polska

Your responsibilities

• Extract, combine, transform, load and cleanse data from various sources to deliver high-quality datasets for analysis, including migration from legacy systems
• Work with data modellers and business analysts to understand the data model and source-to-staging mapping
• Create analysis-ready data structures, including DDLs for MS Cloud SQL Server tables and views in the Pond database (a subset of the Data Lake)
• Design and develop Databricks notebooks that extract data from input files and join related entities into Shared File Formats (SFFs)
• Configure Ingestion Framework tables to load staging and Data Lake Core (DLC) tables
• Build large-scale batch and real-time data pipelines in cloud or on-premises environments and develop performance-efficient algorithms
• Apply CI/CD best practices and introduce automation in existing and future integration flows
• Track data consumption patterns and work with data governance and information stewards on the curated data catalogue
• Drive improvements in data acquisition, testing and tooling to raise data quality
• Monitor performance and optimise SQL queries
• Prepare detailed design documents based on business and system requirement documents
• Work in cross-functional agile teams and take part in requirement analysis and estimates

Our requirements