Lead Software Engineer- Data Engineer

Career Area: Technology, Digital and Data Job Description: Your Work Shapes the World at Caterpillar Inc. When you join Caterpillar, you're joining a global team who cares not just about the work we do but also about each other. We are the makers, problem solvers, and future world builders who are creating stronger, more sustainable communities. We don't just talk about progress and innovation here we make it happen, with our customers, where we work and live. Together, we are building a better world, so we can all enjoy living in it. Software Engineering Lead Data Platform About the Role We are seeking a highly skilled Software Engineering Lead Data Platform to lead the development of Caterpillars next-generation Digital Manufacturing Data Platform. This platform enables large-scale data ingestion, transformation, and analytics across manufacturing, supply chain, and engineering ecosystems. The ideal candidate will bring deep expertise in data engineering, large-scale data ingestion, AWS-based architectures, and Snowflake, along with strong leadership and software engineering discipline. Key Responsibilities Leadership & DeliveryLead and mentor a team of data engineers and platform developersDrive Agile execution and ensure predictable, high-quality deliveryEstablish engineering best practices, code quality, and CI/CD standards Data Platform & ArchitectureArchitect scalable and secure data platforms on AWSDesign robust data ingestion frameworks for batch and near real-time pipelinesDefine best practices in data modeling, governance, and metadata managementData Engineering & IngestionLead design and development of scalable ingestion pipelines (structured and unstructured data)Build and optimize Snowflake-based data platforms for performance and costEnable ingestion of diverse sources (databases, APIs, files, streaming data) Cloud & Platform EngineeringLeverage AWS services (S3, Glue, Lambda, EMR, Redshift, etc.) for end-to-end pipelinesImplement CI/CD pipelines using Azure DevOps / JenkinsEnsure system scalability, resiliency, and operational readinessSoftware Engineering ExcellenceEnforce software engineering principles (modular design, code quality, testing, version control)Drive automation and continuous improvementPromote reusable frameworks for ingestion and transformation Stakeholder CollaborationPartner with product managers, SMEs, and business stakeholdersTranslate business needs into scalable data solutionsMust-Have Skills 10+ years of experience in Data Engineering / Data Platform rolesStrong experience in AWS data ecosystem (S3, Glue, Lambda, EMR, Redshift)Deep expertise in Snowflake (architecture, optimization, data modeling)Strong programming skills in Python and SQLExtensive experience with data ingestion pipelines and ETL/ELT frameworksExposure to real-time streaming (Kafka, Spark Streaming)Experience with CI/CD tools (GitHub, Jenkins, AWS CloudFormation etc.)Solid understanding of distributed systems and scalable architecturesStrong foundation in software engineering principles (Git, testing, design patterns)Experienced in working with Agile teamsCollaborate with Data Science and AI teams to operationalize ML models and analytics workflows.Promote integration of AI capabilities into data engineering pipelines (e.g., GenAI, MCP, ATA).Support real-time analytics and edge AI use cases in manufacturing environments.Use AI extensively in building and testing Data Ingestion and Data pipelineNice-to-Have Skills Experience with Graph Databases (Neo4j, Neptune)Experience with Vector Databases (Milvus, OpenSearch)Knowledge of NVIDIA ecosystem and RAPIDS (cuDF, cuML, cuGraph)Experience integrating AI/ML pipelines or GenAI workflowsQualifications Bachelors or Masters degree in Computer Science / EngineeringProven track record in leading engineering teams and delivering at scaleStrong leadership, communication, and stakeholder management skills This position requires working onsite five days a week. Relocation is available for this position. Posting Dates: August 7, 2026 - August 20, 2026 Caterpillar is an Equal Opportunity Employer. Qualified applicants of any age are encouraged to apply Not ready to apply Join our Talent Community. .