Sr. Data Engineering Specialist (AWS)

This role drives the development and implementation of modern data engineering practices, policies, and cloud-native solutions to support Wealth Management’s DataOps platform on AWS. The specialist will manage the full data lifecycle, ensuring secure, efficient, and cost-effective collection, transformation, storage, and access of data across hybrid and cloud-native environments.

As a senior member of the Wealth Data Engineering team, the role integrates data from legacy investment platforms and new cloud-based systems to build a unified, governed data tier. It applies DataOps and DevOps principles using AWS-native services to enable real-time insights, automation, and reliable delivery of curated datasets for analytics, ML, and regulatory use cases.

***This is a HYBRID role****

Main Responsibilities :

• Design and implement secure, scalable, and cost-efficient data pipelines using AWS Glue, Step Functions, Apache Airflow and Lambda to support batch and streaming ingestion from cloud native, third party (e.g. Snowflake) , and on-premises; relational and/or non-relational data sources

• Build and maintain a centralized data lake using AWS foundational components such as S3, Lake Formation, and Apache Iceberg to enable federated access and fine-grained security across Wealth business domains.

• Integrate data from on-premises and cloud systems using AWS DMS, Glue connectors, and Redshift federated queries.

• Design CI/CD pipelines for infrastructure-as-code and data deployments using AWS CloudFormation, CDK, CodePipeline, and/or Terraform.

• Monitor data health, quality, and lineage using AWS Glue Data Catalog, CloudWatch, and integrate with third-party observability tools like Monte Carlo or OpenMetadata.

• Collaborate with data scientists, analysts, and product owners to ensure curated datasets are discoverable, trustworthy, and reusable across analytics and machine learning use cases.

• Define and implement data quality rules, metadata capture, and schema versioning for curated assets using AWS Glue DataBrew and Data Catalog.

• Build AWS-native services and solutions to support governance, access control, encryption, and auditing to meet financial industry compliance standards.

• Translate Wealth business requirements into data models using Redshift, Aurora (PostgreSQL), and DynamoDB for varied analytical and operational workloads.

• Champion DataOps principles by building reusable components, self-service capabilities, and automated testing, version control, and deployment processes for data assets.

• Creates and maintains an optimal data pipeline architecture, in AWS Cloud and hybrid infrastructure.

• Contribute to container orchestration (ECS, Fargate) and data lake security (Lake Formation).

• Builds analytics tools that utilize the data pipeline to provide actionable insights into customer, operational efficiency and other key business performance metrics, using AWS Redshift and Power BI.

• Works with experts in data strategy and data governance to suggest quality enhancements.

• Creates and maintains various applications using different types of programming languages.

• Designs, configures and implements the cloud-native software that solves actual business problems using cloud technology.

• Creates instructions for cloud computing operations and documentation.

• Establishes a development environment and continuous integration pipeline. Utilizes modern application patterns and code efficiency.

• Translates user needs into technical specifications by understanding, conceptualizing, and facilitating technical requirements from users.

• Operates at a group/enterprise-wide level and serves as a specialist resource to senior leaders and stakeholders.

• Applies expertise and thinks creatively to address unique or ambiguous situations and to find solutions to problems that can be complex and non-routine.

• Implements changes in response to shifting trends.

• Broader work or accountabilities may be assigned as needed.

Technical Qualifications:

- Min of 7+ years of experience in data engineering or cloud-based data platform roles with a minimum 5 years of AWS data solutions experience

- 7+ years of experience with each of the following skills/areas:

• AWS Services: S3, Glue, Redshift, Lake Formation, Lambda, Step Functions, CloudFormation, CodePipeline, IAM, DMS, Athena, CloudWatch, DataBrew, Data Catalog.

• SQL and NoSQL database design (Redshift, Aurora PostgreSQL, DynamoDB).

• Metadata management, data lineage, and data observability tooling.

• Infrastructure-as-Code and DevOps tools: Terraform, CDK, Git, Jenkins, CodeBuild.

• Python, PySpark, SQL, and shell scripting for data engineering.

• Familiarity with ML workflows on Amazon SageMaker is a plus.

Methodologies (Intermediate to Advanced): 5-7+ yrs of experience in each of the following areas:

• DataOPs, MLOPs and DevOps principles applied to data engineering.

• Knowledge of financial services data models and regulatory compliance in wealth management (e.g., KYC, suitability, client reporting).

Core Skills:

• Excellent problem-solving, communication, and stakeholder engagement abilities.

• Ability to manage complexity, ambiguity, and shifting priorities across domains.

• Strong time management

• Proven ability to work with teams across the organization (both horizontally and vertically)