01Responsibilities
GenAI Application Design & Implementation:Design and implement comprehensive AI evaluation and experimentation frameworks for LLMs, vision models, RAG pipelines, and agentic workflows.Build automated evaluation systems to assess model outputs for accuracy, relevance, bias, hallucinations, safety, and regression stability.Develop quality benchmarks and continuous testing pipelines covering content quality, safety, alignment, and enterprise compliance.Establish AI-specific quality gates and acceptance criteria integrated into Agile sprints and CI/CD pipelines.Design, develop, and evaluate data pipelines and RAG workflows using Promptflow, Azure AI Search, ADF Pipelines, Databricks, Spark, and Vector Databases.Validate prompt engineering strategies, prompt consistency, and inference pipelines using GenAI-specific testing tools.Perform prompt-based scenario testing, hallucination detection, and regression validation across model versions.System Support & Operational Excellence:Develop and maintain monitoring capabilities for model drift detection, data quality validation, inference latency, and system reliability.Monitor production GenAI systems using Azure Application Insights, Dynatrace, and AI evaluation platforms.Apply AI risk-focused testing, including bias, fairness, safety, adversarial testing, and red-teaming methodologies.Ensure optimized utilization of infrastructure and compute across diverse AI workloads.Automation & Test Engineering:Build and maintain test automation frameworks (primarily in Python) for AI/ML evaluation and end-to-end workflow validation.Write automation scripts to simulate user behavior and backend interactions.Perform exploratory testing, regression testing, and end-to-end system validation for AI-enabled applications.Track and manage defects using AI evaluation platforms and Agile tools; document test plans, execution results, and evaluation metrics.Mentorship & Team Leadership:Lead design and evaluation reviews; mentor junior engineers and evaluation specialists.Collaborate with product managers to convert business and regulatory requirements into test cases, benchmarks, and test data.Drive innovation by adopting emerging GenAI evaluation techniques and aligning AI initiatives with enterprise goals.Foster a culture of technical excellence, responsible AI, and continuous improvement.Essential Skills:Design and implementation of AI evaluation frameworks for LLMs, RAG, vision models, and agentic systemsGenerative AI engineering: prompt engineering, model integration, inference pipelinesData pipelines and RAG workflows (Promptflow, Azure AI Search, ADF Pipelines, Vector DBs)Automated testing and CI/CD integration for AI systemsMonitoring and observability (Azure Application Insights, Dynatrace)AI risk, safety, bias, fairness, hallucination detection, and adversarial testingLarge-scale experimentation and performance optimizationStrong mentorship, technical leadership, and cross-team collaborationMust Have Skills:Generative AI Azure (AI, Data, and Monitoring services)AI/ML Platform tools (Databricks, Azure AI/ML, GCP Vertex, AWS Bedrock)Python (test automation, evaluation frameworks)Software Engineering & Test Engineering fundamentalsBasic Qualifications:1-3 years of experience in evaluation or test engineering with a strong focus on AI/ML and Generative AI systems.Hands-on experience evaluating LLMs, RAG pipelines, and AI-powered applications in production environments.Strong understanding of AI/ML evaluation concepts including accuracy, relevance, regression, latency, and stability.Bachelors Degree in Computer Science, Data Science, Artificial Intelligence, or equivalent practical experience.Preferred Qualifications:Knowledge of AI risks including bias, fairness, safety, hallucinations, and adversarial attacks.Experience in highly regulated or governed AI environments.Strong collaboration skills with data scientists, ML engineers, platform teams, and product managers.Passion for AI quality, responsible AI practices .
