01Key Responsibilities
Design, develop, and lead production-grade data engineering applications across agile teams, combining hands-on coding with technical leadership, design reviews, and end-to-end delivery ownership. Design and implement scalable ELT/ETL and 24x7 real-time event streaming pipelines using Ab Initio Continuous Flows/Spark Streaming and Kafka ecosystems (AWS MSK and Confluent Kafka), ensuring fault tolerance, strong delivery guarantees (idempotency/exactly-once where applicable), and near-zero downtime deployments. Own event-driven data workflows that process high-throughput, low-latency financial transactions across mission-critical systems, and build/operate Lakehouse data infrastructure on AWS (S3, Glue, EMR, Redshift), using Ab Initio (must-have) and Spark as core engineering tool. Develop AI-ready data products and feature pipelines for downstream analytics and operational AI use cases, and lead cloud migration efforts to modernize legacy batch ETL into cloud-native, event-driven architecture. Collaborate with solution architects to translate designs into robust, scalable implementations; partner with product owners and program managers in a SAFe/Scaled Agile environment to align delivery with business priorities. Define and enforce data engineering standards, coding best practices, CI/CD, and DevOps processes; mentor engineers and foster a culture of engineering excellence and continuous learning. Required Skills/Knowledge : Bachelors degree in Computer Science, Engineering, or related field with 8+ years of experience in ELT/ETL, data warehousing, or Lakehouse engineering; or, in lieu of a degree with 10+ years of equivalent hands-on experience. 8+ years of full lifecycle data engineering experience (design, development, testing, documentation, performance tuning, and production support) across large-scale technologies including Ab Initio, Kafka (MSK/Confluent), Spark, Hive, RDBMS (Oracle/MySQL), and AWS services (EMR, MSK, Glue, S3, Redshift, IAM). 5+ years of experience with 24x7 real-time/streaming pipelines and 2+ years with AWS cloud platform engineering, including infrastructure-as-code and containerized workloads. Experience leading or mentoring data engineering teams in an Agile/SAFe delivery environment. Ability to travel for business as needed. Desired Skills/Knowledge : Proven experience as a hands-on data engineering technical lead in a Protected environment, owning mission-critical data platform applications with end-to-end accountability across batch and real-time pipelines in a DevOps model. Experience in FinTech, credit card, or banking environments handling sensitive data in a regulated context, with the ability to communicate complex technical topics and influence decisions across IT and business stakeholders. AWS Solutions Architect or AWS Data Engineer certification (or actively pursuing). Exposure to agentic
