Lead AWS Data Engineer

Role Summary The Lead AWS Data Engineer provides

technical leadership and hands‑on execution

for enterprise data platforms hosted on

Amazon Web Services (AWS)

. This role leads the design, migration, modernization, and operation of

cloud‑native data architectures

supporting mission‑critical financial, advisor payout, mobility, and corporate analytics platforms. The Lead AWS Data Engineer owns

end‑to‑end technical decision‑making

for AWS data platforms, including architecture, security, orchestration, and production readiness. The role requires deep expertise in

AWS data services, Python and PySpark development, workflow orchestration, data lake design, infrastructure as code, and operational excellence

, along with the ability to mentor engineers and partner effectively with infrastructure, IAM, and database teams.

Key Responsibilitie sTechnical Leadership & Platform Ownershi pServe as the

technical lead and design authorit

y for AWS data engineering initiatives across multiple enterprise platforms .Own architectural decisions related to scalability, reliability, security, and cost optimization of AWS data platforms .Define and enforce engineering standards, coding patterns, and operational best practices for cloud data pipelines .Provide hands‑on technical guidance, design reviews, and code reviews for data engineers

. AWS Data Platform Engineeri ngLead the design, development, and support of cloud‑native data pipelines usin

g Amazon S3, AWS Glue (PySpark), MWAA (Apache Airflow), and AWS Step Functio

n s.Driv

e on‑premises to AWS data platform migratio

ns, including reverse engineering of legacy ETL workflows and re‑implementation using AWS‑native service s.Re‑architect legac

y Oracle Data Integrator (OD

I)–based ETL processes int

o scalable PySpark‑based Glue jo

b s.Optimize Spark workloads for performance, memory usage, and cost efficiency in AWS Glue environment

s. Data Lake, Iceberg & Architecture Des ignArchitect and implement enterpri

se AWS data lakes using Medallion architecture (Bronze, Silver, Go

l d).Design and mana

ge Apache Iceberg tab

les to support incremental processing, schema evolution, and efficient data lake operatio ns.Establish standardized ingestion, transformation, and consumption patterns across financial, mobility, and corporate datase ts.Ensure data quality, reconciliation, lineage, and auditability across all layers of the data platfo

rm. Workflow Orchestration & Automa tionLead orchestration strategy us

ing MWAA (Managed Workflows for Apache Airf

l ow).Design and implement Airflow DAGs in Python to orchestrate end‑to‑end workflows, including Glue jobs, validations, and downstream dependenc ies.Implement scheduling, retry logic, monitoring, and failure handling to ensure resilient and scalable pipeli nes.Integrate orchestration workflows with AWS services such

as S3, Glue, Athena, Iceberg‑based data l

akes, and downstream syst

ems. Security, Infrastructure & AWS Netwo rkingDrive implementation of AWS security best practices, inclu

ding IAM role design, least‑privilege access, encryption using AWS KMS, and secrets manag

e ment.Lead configuration of AWS networking components suc

h as VPC Endpoints (

VPCE) to enable secure service‑to‑service communica tion.Manage infrastructure provisioning u

sing Terr

aform, ensuring repeatable and auditable deployments across DEV, QA, and PROD environm ents.Coordinate with IAM, network, DevOps, and DBA teams to resolve access, firewall, and Oracle database connectivity challe

nges. Production Readiness, Operations & S uppo

rtOwn production rea

diness for AWS data platforms, including configuration, secrets, access controls, and deployment pla nning.Act as the escalation point for complex production issues, performing root‑cause analysis and permanent fixes.Implement logging, metrics, and alerting

using Amazon Clou

dWatch to meet enterprise SLAs and availability ta rgets.Support parallel‑run and hybrid architectures during migration phases to ensure business conti

nuity. Automation, Compliance & Regulatory Ena blementDesign and o

versee Python‑based automation so

lutions supporting operational efficiency and compliance initiatives (e.g., file retention and document proce ssing).Ensure pipeline design

s meet regulatory, audit, and enterprise governance requi

rements, including traceability and controlled data ha

ndling. Collaboration & Stakeholder En gagementPartner closely with data architects, DevOps teams, infrastructure teams, and business stakeholders to deliver AWS data solutions aligned with enterprise s business and platform requirements into scalable technical designs and executio n plans.Produce technical documentation and support knowledge transfer to enable long‑term platform sustain

ability. Required Quali ficationsExperie

nce with Apach

e Iceberg or similar data lake table to analytics or BI platforms (e.g., ThoughtSpot, Tableau).Exposure to Oracle databases or legacy ETL tools (e. g., ODI).Experience in financial services or regulated enterprise envi ronments.Familiarity with CI/CD practices for data engineering w

orkloads.