01Overview
At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless workbut its work worth doing. If youre driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.
Test Architect / Tech Lead - CTI MD Tech (GRA/GSC)
Global Regulatory Affairs (GRA) Global Scientific Communications (GSC) CTI-MD Tech@Lilly
Global Regulatory Affairs (GRA) and Global Scientific Communications (GSC) together own how Lilly's medicines move through health authority review, stay compliant once approved, and get communicated externally spanning regulatory submissions, labelling, clinical trial disclosures, and scientific publications. Both functions operate under strict compliance and audit-readiness requirements, and are scaling agentic and generative AI capabilities across their platforms, creating a growing need for consistent, reusable test automation and AI quality engineering practices across GRA and GSC.
The Test Architect / Tech Lead (R4), GRA/GSC is accountable for building the test automation frameworks and reusable testing components that drive quality engineering across GRA and GSC, with particular depth in evaluating and validating generative AI and agentic systems. The role also carries tech lead accountability: designing and building the application architecture for GRA/GSC platforms and owning delivery of key initiatives end to end, not solely test strategy. This is a single R4 path role by design: test standards, automation architecture, and application architecture need to be set once and held consistently rather than fragmented across roles without the seniority to drive strategic evolution end to end. The role owns the build-out of reusable frameworks and components, drives test automation initiatives, leads delivery of key technical initiatives, and manages stakeholders to ensure quality and AI evaluation rigor scale consistently as GRA/GSC's agentic platforms mature.
Path/Level: R4 (Test Architect / Tech Lead)
Key ResponsibilitiesReusable Test Framework Development design, build, and own reusable test automation frameworks and shared components (libraries, harnesses, fixtures, evaluation pipelines) that can be adopted consistently across GRA and GSC rather than rebuilt per program.Application Architecture & Design design and build the application architecture for GRA/GSC platforms as a tech lead defining technical approach, component design, and integration patterns alongside the test and quality strategy.Delivery Ownership act as tech lead for key technical initiatives, owning end-to-end delivery from architecture and build through testing and release and holding accountability for outcomes, not just quality gates.Automation Enablement drive test automation efforts across GRA and GSC, identifying opportunities to replace manual and duplicated testing effort with shared, scalable automation.AI & Agentic Quality Engineering build and own evaluation frameworks for generative AI and agentic systems groundedness validation, hallucination detection, relevance/accuracy testing, and RAG and LLM evaluation using tools such as RAGAS, DeepEval, and LLM-as-a-Judge approaches.Test Initiative Leadership define and drive test initiatives end to end scoping, prioritizing, and delivering automation and quality-engineering improvements across concurrent teams and workstreams.Stakeholder Management partner with engineering leads, product owners, and quality/compliance stakeholders to align on architecture, delivery plans, test strategy, and adoption of shared frameworks.Observability & Trace Analysis establish agent observability and trace-analysis practices to diagnose failures in multi-agent and agentic workflows, and feed findings back into architecture, framework, and test design.Synthetic Test Data & Coverage Strategy build synthetic test data generation capabilities and apply risk-based prioritization to determine test depth and coverage across critical workflows.Talent Development & Mentorship mentor test engineers and technical leads, building a team capable of operating independently on architecture, framework design, automation, and AI evaluation.Technical & Strategic Expertise AreasApplication & Solution Architecture AI Evaluation Frameworks (RAGAS, DeepEval, LLM-as-a-Judge) Groundedness Validation Hallucination Detection Relevance & Accuracy Testing Generative AI Testing Agentic AI Testing RAG Validation LLM Evaluation Prompt Engineering & Context Validation Multi-Agent Systems .
