Senior Data Engineer to build and maintain data pipelines on the Databricks Lakehouse Platform for the financial services industry

We are looking for a Senior Data Engineer to build and maintain data pipelines on the Databricks Lakehouse Platform for the financial services industry

Full Time Permanent Role

2 times a week on site (Near King Station)

Must Haves Skills:

• Bachelor’s degree in computer science, Engineering, or related field
• 4+ years of experience in data engineering
• 1+ years of hands-on experience with Databricks platform
• Strong programming skills in Python
• Experience with Spark and distributed computing
• Working knowledge of AWS services (S3, Glue, Lambda)
• Experience with Delta Lake and Lakehouse architecture
• Familiarity with data modelling and SQL
• Understanding of ETL/ELT principles and patterns
• Experience with version control systems (Git)
• Experience with CI/CD for data pipelines
• Familiarity with Agile development methodologies
• Experience with real-time data processing is a plus

Job Duties in Brief:

• Participate and contribute on all team activities such as Sprint Planning, Sprint Execution, Daily Scrum
• Develop and maintain data pipelines using Databricks Lakehouse and Delta Lake
• Implement ETL/ELT workflows using Spark (Python) in Databricks environment
• Work with AWS services (S3, Glue) for data lake storage and catalog management
• Create and optimize Spark jobs for efficient data processing and cost management
• Build and maintain data quality checks and monitoring systems
• Configure and manage Databricks notebooks and jobs
• Implement proper security and access controls using Unity Catalog
• Participate in code reviews and documentation efforts
• Stay current with Databricks features and data engineering best practices
• Support real-time data processing using structured streaming when required