Data Engineer to build and maintain data pipelines on the Databricks Lakehouse Platform for the financial services industry

Full Time role

2 times a week on site (Near King Station)

Must Haves:

· Bachelor’s degree in computer science, Engineering, or related field

· 4+ years of experience in data engineering

· 1+ years of hands-on experience with Databricks platform

· Strong programming skills in Python

· Experience with Spark and distributed computing

· Working knowledge of AWS services (S3, Glue, Lambda)

· Experience with Delta Lake and Lakehouse architecture

· Familiarity with data modelling and SQL

· Understanding of ETL/ELT principles and patterns

· Experience with version control systems (Git)

· Good communication and collaboration abilities

· Experience with CI/CD for data pipelines

· Familiarity with Agile development methodologies

· Experience with real-time data processing is a plus

· Self-motivated with ability to work independently

Job Duties in Brief:

· Participate and contribute on all team activities such as Sprint Planning, Sprint Execution, Daily Scrum

· Develop and maintain data pipelines using Databricks Lakehouse and Delta Lake

· Implement ETL/ELT workflows using Spark (Python) in Databricks environment

· Work with AWS services (S3, Glue) for data lake storage and catalog management

· Create and optimize Spark jobs for efficient data processing and cost management

· Build and maintain data quality checks and monitoring systems

· Configure and manage Databricks notebooks and jobs

· Implement proper security and access controls using Unity Catalog

· Participate in code reviews and documentation efforts

· Stay current with Databricks features and data engineering best practices

· Support real-time data processing using structured streaming when required