Designs and implements Azure Databricks and Azure Data Factory pipelines and SQL data models for our clients in the banking industry. This role will involve developing new integrations with data sources our clients leverage, using a mixture of Azure Databricks, Azure Data Factory, and Azure SQL, while also deploying existing code into new client environments using similar tools.
Creative problem solving is essential in this position to quickly address changing client needs and adapt to an evolving dataset we have at our disposal. Must be legally authorized to work in the United States.
- Develop workbooks in Azure Databricks that manages the movement of data from source through to the eventual data warehouse environment.
- Design Azure Data Factory pipelines to manage the execution of Databricks workbooks and additional code that may reside in Azure SQL
- Design, test, and implement stored procedure in Azure SQL or Azure Synapse to move data from a staging environment and/or operational data store into the dimensional data model.
- Extend existing tables to add new columns and calculations based on client feedback.
- Leverages existing data infrastructure to fulfill all data-related requests, perform necessary data housekeeping, data cleansing, normalization, hashing, and implementation of required data model changes.
- Troubleshoots problems, identifies possible solutions, and resolves accordingly.
- Work with business counterparts to define requirements for data desired to be provisioned updating defined requirement document templates in partnership with business counterpart. Work with the manager to transform business requirements into appropriate schema and data model.
- Daily tasks can include:
- Use ADF/Databricks to transform data within the data warehouse.
- Tune databases and ETL for maximum performance.
- Data Profiling, Data Cleansing, and Data Auditing.
- Loading large volumes of data.
- Decoding and writing complex SQL queries.
- Performance tuning of queries and data loading process.
- Modeling data into dimensional model structures.