LLM Evaluation Engineer

About the RoleWe are looking for an LLM Evaluation Engineer to design, build, and support automated and human evaluation frameworks that measure the accuracy, safety, reliability, and business performance of Large Language Model (LLM) applications. The ideal candidate will collaborate with business, data, and engineering teams to deliver secure, scalable, and measurable AI solutions by establishing robust evaluation methodologies and continuous quality improvement processes. Key ResponsibilitiesDesign and develop evaluation frameworks for Large Language Models (LLMs) and Generative AI applications.Create and maintain high-quality evaluation datasets, benchmarks, and test scenarios.Implement LLM-as-Judge methodologies to automate response quality assessment.Define and measure key evaluation metrics such as accuracy, relevance, faithfulness, safety, latency, and business KPIs.Build automated evaluation pipelines and integrate them into AI development workflows.Conduct human evaluations and analyze qualitative feedback to improve model performance.Track experiments, compare model versions, and maintain reproducible evaluation results.Collaborate with AI engineers, data scientists, and business stakeholders to define evaluation criteria and success metrics.Monitor production model performance and recommend improvements based on evaluation insights.Document evaluation methodologies, metrics, and governance best practices.Required SkillsExperience designing evaluation datasets and benchmark suitesHands-on experience with LLM-as-Judge evaluation techniquesStrong understanding of LLM evaluation metrics and AI quality measurementProficiency in PythonExperience with experiment tracking and model comparison toolsKnowledge of Generative AI, Large Language Models (LLMs), and AI evaluation frameworksStrong analytical and statistical reasoning skillsExperience RequirementsUp to 5 years of overall experienceMinimum 12 years of relevant hands-on experience in LLM evaluation, AI quality assurance, Generative AI, NLP, or related technologies Skills Required Llm .