QA - AI Evaluator

About the Role

The QA AI Evaluator ensures quality, reliability, governance, and business effectiveness of Agentic AI solutions. Responsibilities include defining evaluation frameworks, validating agent accuracy/reasoning, assessing prompts/LLM outputs for hallucination risks, benchmarking models/tools, ensuring Responsible AI/security compliance, providing insights/scorecards to leadership, measuring AI metrics (accuracy/relevance/completeness), identifying risk gaps, supporting adoption/change management, and collaborating with cross-functional teams.

Requirements

• Strong experience in AI testing/QA/LLM evaluation or agentic AI assurance.
• Deep understanding of LLMs, prompt engineering, RAG, agents, multi-agent architectures.
• Expertise in defining frameworks/metrics/KPIs/scorecards/benchmarking.
• Experience validating accuracy/reasoning/reliability/hallucinations/business outcomes.
• Knowledge of Responsible AI/Security/Privacy/Governance/Risk/Compliance practices.
• Analytical/problem-solving/root cause analysis skills with data-driven assessment experience.

Essential

• 3-4 years in AI testing/QA/LLM evaluation or agentic AI assurance.
• Solid understanding of LLMs, prompt engineering, RAG, agents/workflows.
• Ability to define frameworks/metrics/KPIs/success criteria.

Desirable

• Demonstrated experience with automation tools and data-driven techniques (3+ years).
• Benchmarking/model optimization exposure.
• Aid adoption/change management/initiative support background.