Job Details:

Position                :                 Conversational AI Model Evaluation Lead

Experience           :                 7+ Years

Location               :                 Noida/Gurugram/Pune/Bengaluru

Work Mode         :                 Hybrid (Shift timings- 01:30 PM to 11:30 PM)

Conversational AI Model Evaluation Lead

We are seeking a highly skilled Conversational AI Model Evaluation Lead with 8–10 years of experience to own evaluation strategy, simulation-led validation, and quality intelligence for enterprise Voice AI, Conversational AI, and LLM-powered agent solutions.

This role combines hands-on model evaluation with lead-level ownership across AI behavior analysis, evaluation framework design, custom metrics, transcript intelligence, and stakeholder-facing insights. The focus is to ensure that conversational AI systems are accurate, grounded, safe, reliable, business-aligned, and continuously improving across real customer journeys.

What You Will Own

1. Conversational AI Evaluation Strategy

• Own the evaluation approach for Conversational AI, Voice AI, virtual agents, agent assist experiences, and LLM-powered conversational workflows.

• Define model assurance strategies that measure AI behavior across intent understanding, entity extraction, response quality, task completion, containment, escalation, and user experience outcomes.

• Partner with product, design, engineering, operations, and client stakeholders to align evaluation criteria with customer journeys, business KPIs, compliance expectations, and release-readiness goals.

• Establish governance for how conversational AI quality is measured, reported, improved, and tracked over time.

2. LLM and AI Behavior Evaluation

• Evaluate LLM-driven responses for groundedness, relevance, completeness, tone, safety, policy adherence, hallucination risk, instruction-following, and contextual consistency.

• Assess NLU/NLP behavior including intent recognition, entity capture, confidence thresholds, disambiguation, fallback behavior, and multi-turn dialog continuity.

• Identify model failure modes such as unsupported answers, hallucinated responses, context loss, repetitive loops, incorrect routing, ambiguous handling, and missed escalation triggers.

• Translate AI behavior observations into clear improvement recommendations for prompt design, knowledge base quality, orchestration logic, model tuning, and conversation design.

3. Simulation-led Validation and Custom Evaluation Metrics

• Design and run simulation-led evaluation programs that replicate realistic user behavior, edge cases, failure scenarios, domain-specific journeys, and high-volume conversational patterns.

• Define custom evaluation metrics across intent accuracy, entity quality, containment, task completion, fallback rate, escalation precision, hallucination risk, response groundedness, conversation efficiency, and customer effort.

• Create evaluation rubrics, scoring models, sampling approaches, benchmark sets, and regression packs for both deterministic conversational flows and generative AI experiences.

• Use the AI agent’s knowledge base, prompts, training data, historical transcripts, and business rules to design realistic evaluation datasets and scenario libraries.

4. Transcript Intelligence and Failure Mode Analysis

• Analyze call transcripts, conversation logs, simulation outputs, user utterances, agent responses, and escalation patterns to identify systemic quality issues.

• Diagnose breakdown points across customer intent, language variation, prompt behavior, knowledge retrieval, dialog flow, tool invocation, integration responses, and handoff logic.

• Develop insights that explain why AI experiences succeed or fail, with clear prioritization based on customer impact, operational impact, business risk, and release readiness.

• Produce structured recommendations for product managers, conversation designers, engineering teams, operations leaders, and client stakeholders.

5. Quality Intelligence, Dashboards, and Stakeholder Reporting

• Build quality intelligence views that track AI performance trends, customer experience risks, defect patterns, release readiness, and improvement opportunities.

• Design dashboards and executive-ready reports that quantify model reliability, containment performance, hallucination risk, task success, customer friction, and business impact.

• Communicate insights clearly to both technical and non-technical audiences, linking model behavior to customer outcomes and operational priorities.

• Enable data-driven prioritization of fixes, prompt improvements, model refinements, knowledge base enhancements, and conversation design updates.

Required Experience

Experience Profile

• 8–10 years of experience across Conversational AI, Voice AI, AI quality engineering, model evaluation, contact center automation, digital CX, or related technology delivery roles.

• Hands-on experience evaluating enterprise conversational systems across healthcare, banking, insurance, utilities, or other customer-service-intensive domains.

• Experience working with AI evaluation platforms, simulation environments, transcript analytics solutions, conversational QA automation tools, or model observability platforms.

• Proven ability to lead evaluation workstreams while also contributing hands-on to evaluation design, scenario creation, analysis, reporting, and improvement recommendations.

• Experience working with US clients or customer journeys serving US markets is highly preferred.

Educational Qualification

• B.Tech / BE (Computer Science, IT, Electronics, or related field)

Core Skills

Technical and AI Evaluation Skills

• Strong understanding of Conversational AI architectures, NLU/NLP concepts, dialog management, knowledge retrieval, prompt behavior, and contact center automation patterns.

• Ability to evaluate LLM-powered experiences for hallucination risk, groundedness, contextual relevance, response quality, instruction adherence, safety, and customer-facing reliability.

• Experience designing evaluation rubrics, custom metrics, scenario libraries, simulation datasets, regression packs, and model quality scorecards.

• Strong analytical capability using Excel, SQL, Python, BI tools, or equivalent data analysis approaches to interpret conversation performance at scale.

• Familiarity with voice AI, telephony flows, speech recognition outputs, transcript analytics, and integration-driven conversational experiences.

Functional and Analytical Skills

• Ability to connect model behavior, conversation defects, and customer experience issues with measurable business impact.

• Strong capability in transcript-driven analysis, failure mode diagnosis, root cause synthesis, and prioritization of improvement opportunities.

• Comfort working across business, product, design, engineering, operations, and client stakeholder groups.

• Ability to create executive-ready narratives that explain quality trends, model risks, release readiness, and recommended actions.

Leadership and Communication Skills

• Lead evaluation planning, quality governance, stakeholder reviews, and cross-functional improvement discussions.

• Mentor team members on evaluation methodology, simulation design, AI behavior analysis, and insight-driven reporting.

• Communicate complex AI quality findings in a clear, structured, and action-oriented manner.

• Operate with strong ownership, attention to detail, curiosity, and a continuous improvement mindset.

Preferred Tools and Platforms

• AI evaluation, simulation, and conversation quality automation platforms

• Transcript analytics, speech analytics, and contact center intelligence tools

• LLM evaluation frameworks, prompt evaluation approaches, and model observability solutions

• Dashboarding and analytics tools such as Power BI, Tableau, Looker, Excel, SQL, Python, or equivalent

• Conversational AI platforms, voice AI platforms, agent assist solutions, and contact center technology ecosystems

Success Measures

• Improved accuracy, groundedness, reliability, and consistency of Conversational AI and LLM-powered experiences

• Reduction in hallucination risk, fallback rates, repeat contacts, avoidable escalations, and unresolved customer journeys

• Improved containment, task completion, intent performance, response relevance, and customer experience outcomes

• Clear visibility into model quality trends, release readiness, defect themes, and improvement priorities

• Actionable insights adopted by product, design, engineering, and client teams to improve AI performance over time