All fields marked * are required
Job Details:
Position : Conversational AI Model Evaluation Lead
Experience : 7+ Years
Location : Noida/Gurugram/Pune/Bengaluru
Work Mode : Hybrid (Shift timings- 01:30 PM to 11:30 PM)
Conversational AI Model Evaluation Lead
We are seeking a highly skilled Conversational AI Model Evaluation Lead with 8–10 years of experience to own evaluation strategy, simulation-led validation, and quality intelligence for enterprise Voice AI, Conversational AI, and LLM-powered agent solutions.
This role combines hands-on model evaluation with lead-level ownership across AI behavior analysis, evaluation framework design, custom metrics, transcript intelligence, and stakeholder-facing insights. The focus is to ensure that conversational AI systems are accurate, grounded, safe, reliable, business-aligned, and continuously improving across real customer journeys.
What You Will Own
1. Conversational AI Evaluation Strategy
Own the evaluation approach for Conversational AI, Voice AI, virtual agents, agent assist experiences, and LLM-powered conversational workflows.
Define model assurance strategies that measure AI behavior across intent understanding, entity extraction, response quality, task completion, containment, escalation, and user experience outcomes.
Partner with product, design, engineering, operations, and client stakeholders to align evaluation criteria with customer journeys, business KPIs, compliance expectations, and release-readiness goals.
Establish governance for how conversational AI quality is measured, reported, improved, and tracked over time.
2. LLM and AI Behavior Evaluation
Evaluate LLM-driven responses for groundedness, relevance, completeness, tone, safety, policy adherence, hallucination risk, instruction-following, and contextual consistency.
Assess NLU/NLP behavior including intent recognition, entity capture, confidence thresholds, disambiguation, fallback behavior, and multi-turn dialog continuity.
Identify model failure modes such as unsupported answers, hallucinated responses, context loss, repetitive loops, incorrect routing, ambiguous handling, and missed escalation triggers.
Translate AI behavior observations into clear improvement recommendations for prompt design, knowledge base quality, orchestration logic, model tuning, and conversation design.
3. Simulation-led Validation and Custom Evaluation Metrics
Design and run simulation-led evaluation programs that replicate realistic user behavior, edge cases, failure scenarios, domain-specific journeys, and high-volume conversational patterns.
Define custom evaluation metrics across intent accuracy, entity quality, containment, task completion, fallback rate, escalation precision, hallucination risk, response groundedness, conversation efficiency, and customer effort.
Create evaluation rubrics, scoring models, sampling approaches, benchmark sets, and regression packs for both deterministic conversational flows and generative AI experiences.
Use the AI agent’s knowledge base, prompts, training data, historical transcripts, and business rules to design realistic evaluation datasets and scenario libraries.
4. Transcript Intelligence and Failure Mode Analysis
Analyze call transcripts, conversation logs, simulation outputs, user utterances, agent responses, and escalation patterns to identify systemic quality issues.
Diagnose breakdown points across customer intent, language variation, prompt behavior, knowledge retrieval, dialog flow, tool invocation, integration responses, and handoff logic.
Develop insights that explain why AI experiences succeed or fail, with clear prioritization based on customer impact, operational impact, business risk, and release readiness.
Produce structured recommendations for product managers, conversation designers, engineering teams, operations leaders, and client stakeholders.
5. Quality Intelligence, Dashboards, and Stakeholder Reporting
Build quality intelligence views that track AI performance trends, customer experience risks, defect patterns, release readiness, and improvement opportunities.
Design dashboards and executive-ready reports that quantify model reliability, containment performance, hallucination risk, task success, customer friction, and business impact.
Communicate insights clearly to both technical and non-technical audiences, linking model behavior to customer outcomes and operational priorities.
Enable data-driven prioritization of fixes, prompt improvements, model refinements, knowledge base enhancements, and conversation design updates.
Required Experience
Experience Profile
8–10 years of experience across Conversational AI, Voice AI, AI quality engineering, model evaluation, contact center automation, digital CX, or related technology delivery roles.
Hands-on experience evaluating enterprise conversational systems across healthcare, banking, insurance, utilities, or other customer-service-intensive domains.
Experience working with AI evaluation platforms, simulation environments, transcript analytics solutions, conversational QA automation tools, or model observability platforms.
Proven ability to lead evaluation workstreams while also contributing hands-on to evaluation design, scenario creation, analysis, reporting, and improvement recommendations.
Experience working with US clients or customer journeys serving US markets is highly preferred.
Educational Qualification
B.Tech / BE (Computer Science, IT, Electronics, or related field)
Core Skills
Technical and AI Evaluation Skills
Strong understanding of Conversational AI architectures, NLU/NLP concepts, dialog management, knowledge retrieval, prompt behavior, and contact center automation patterns.
Ability to evaluate LLM-powered experiences for hallucination risk, groundedness, contextual relevance, response quality, instruction adherence, safety, and customer-facing reliability.
Experience designing evaluation rubrics, custom metrics, scenario libraries, simulation datasets, regression packs, and model quality scorecards.
Strong analytical capability using Excel, SQL, Python, BI tools, or equivalent data analysis approaches to interpret conversation performance at scale.
Familiarity with voice AI, telephony flows, speech recognition outputs, transcript analytics, and integration-driven conversational experiences.
Functional and Analytical Skills
Ability to connect model behavior, conversation defects, and customer experience issues with measurable business impact.
Strong capability in transcript-driven analysis, failure mode diagnosis, root cause synthesis, and prioritization of improvement opportunities.
Comfort working across business, product, design, engineering, operations, and client stakeholder groups.
Ability to create executive-ready narratives that explain quality trends, model risks, release readiness, and recommended actions.
Leadership and Communication Skills
Lead evaluation planning, quality governance, stakeholder reviews, and cross-functional improvement discussions.
Mentor team members on evaluation methodology, simulation design, AI behavior analysis, and insight-driven reporting.
Communicate complex AI quality findings in a clear, structured, and action-oriented manner.
Operate with strong ownership, attention to detail, curiosity, and a continuous improvement mindset.
Preferred Tools and Platforms
AI evaluation, simulation, and conversation quality automation platforms
Transcript analytics, speech analytics, and contact center intelligence tools
LLM evaluation frameworks, prompt evaluation approaches, and model observability solutions
Dashboarding and analytics tools such as Power BI, Tableau, Looker, Excel, SQL, Python, or equivalent
Conversational AI platforms, voice AI platforms, agent assist solutions, and contact center technology ecosystems
Success Measures
Improved accuracy, groundedness, reliability, and consistency of Conversational AI and LLM-powered experiences
Reduction in hallucination risk, fallback rates, repeat contacts, avoidable escalations, and unresolved customer journeys
Improved containment, task completion, intent performance, response relevance, and customer experience outcomes
Clear visibility into model quality trends, release readiness, defect themes, and improvement priorities
Actionable insights adopted by product, design, engineering, and client teams to improve AI performance over time
Required
Preferred
All fields marked * are required
Job Details:
Position : Conversational AI Model Evaluation Lead
Experience : 7+ Years
Location : Noida/Gurugram/Pune/Bengaluru
Work Mode : Hybrid (Shift timings- 01:30 PM to 11:30 PM)
Conversational AI Model Evaluation Lead
We are seeking a highly skilled Conversational AI Model Evaluation Lead with 8–10 years of experience to own evaluation strategy, simulation-led validation, and quality intelligence for enterprise Voice AI, Conversational AI, and LLM-powered agent solutions.
This role combines hands-on model evaluation with lead-level ownership across AI behavior analysis, evaluation framework design, custom metrics, transcript intelligence, and stakeholder-facing insights. The focus is to ensure that conversational AI systems are accurate, grounded, safe, reliable, business-aligned, and continuously improving across real customer journeys.
What You Will Own
1. Conversational AI Evaluation Strategy
Own the evaluation approach for Conversational AI, Voice AI, virtual agents, agent assist experiences, and LLM-powered conversational workflows.
Define model assurance strategies that measure AI behavior across intent understanding, entity extraction, response quality, task completion, containment, escalation, and user experience outcomes.
Partner with product, design, engineering, operations, and client stakeholders to align evaluation criteria with customer journeys, business KPIs, compliance expectations, and release-readiness goals.
Establish governance for how conversational AI quality is measured, reported, improved, and tracked over time.
2. LLM and AI Behavior Evaluation
Evaluate LLM-driven responses for groundedness, relevance, completeness, tone, safety, policy adherence, hallucination risk, instruction-following, and contextual consistency.
Assess NLU/NLP behavior including intent recognition, entity capture, confidence thresholds, disambiguation, fallback behavior, and multi-turn dialog continuity.
Identify model failure modes such as unsupported answers, hallucinated responses, context loss, repetitive loops, incorrect routing, ambiguous handling, and missed escalation triggers.
Translate AI behavior observations into clear improvement recommendations for prompt design, knowledge base quality, orchestration logic, model tuning, and conversation design.
3. Simulation-led Validation and Custom Evaluation Metrics
Design and run simulation-led evaluation programs that replicate realistic user behavior, edge cases, failure scenarios, domain-specific journeys, and high-volume conversational patterns.
Define custom evaluation metrics across intent accuracy, entity quality, containment, task completion, fallback rate, escalation precision, hallucination risk, response groundedness, conversation efficiency, and customer effort.
Create evaluation rubrics, scoring models, sampling approaches, benchmark sets, and regression packs for both deterministic conversational flows and generative AI experiences.
Use the AI agent’s knowledge base, prompts, training data, historical transcripts, and business rules to design realistic evaluation datasets and scenario libraries.
4. Transcript Intelligence and Failure Mode Analysis
Analyze call transcripts, conversation logs, simulation outputs, user utterances, agent responses, and escalation patterns to identify systemic quality issues.
Diagnose breakdown points across customer intent, language variation, prompt behavior, knowledge retrieval, dialog flow, tool invocation, integration responses, and handoff logic.
Develop insights that explain why AI experiences succeed or fail, with clear prioritization based on customer impact, operational impact, business risk, and release readiness.
Produce structured recommendations for product managers, conversation designers, engineering teams, operations leaders, and client stakeholders.
5. Quality Intelligence, Dashboards, and Stakeholder Reporting
Build quality intelligence views that track AI performance trends, customer experience risks, defect patterns, release readiness, and improvement opportunities.
Design dashboards and executive-ready reports that quantify model reliability, containment performance, hallucination risk, task success, customer friction, and business impact.
Communicate insights clearly to both technical and non-technical audiences, linking model behavior to customer outcomes and operational priorities.
Enable data-driven prioritization of fixes, prompt improvements, model refinements, knowledge base enhancements, and conversation design updates.
Required Experience
Experience Profile
8–10 years of experience across Conversational AI, Voice AI, AI quality engineering, model evaluation, contact center automation, digital CX, or related technology delivery roles.
Hands-on experience evaluating enterprise conversational systems across healthcare, banking, insurance, utilities, or other customer-service-intensive domains.
Experience working with AI evaluation platforms, simulation environments, transcript analytics solutions, conversational QA automation tools, or model observability platforms.
Proven ability to lead evaluation workstreams while also contributing hands-on to evaluation design, scenario creation, analysis, reporting, and improvement recommendations.
Experience working with US clients or customer journeys serving US markets is highly preferred.
Educational Qualification
B.Tech / BE (Computer Science, IT, Electronics, or related field)
Core Skills
Technical and AI Evaluation Skills
Strong understanding of Conversational AI architectures, NLU/NLP concepts, dialog management, knowledge retrieval, prompt behavior, and contact center automation patterns.
Ability to evaluate LLM-powered experiences for hallucination risk, groundedness, contextual relevance, response quality, instruction adherence, safety, and customer-facing reliability.
Experience designing evaluation rubrics, custom metrics, scenario libraries, simulation datasets, regression packs, and model quality scorecards.
Strong analytical capability using Excel, SQL, Python, BI tools, or equivalent data analysis approaches to interpret conversation performance at scale.
Familiarity with voice AI, telephony flows, speech recognition outputs, transcript analytics, and integration-driven conversational experiences.
Functional and Analytical Skills
Ability to connect model behavior, conversation defects, and customer experience issues with measurable business impact.
Strong capability in transcript-driven analysis, failure mode diagnosis, root cause synthesis, and prioritization of improvement opportunities.
Comfort working across business, product, design, engineering, operations, and client stakeholder groups.
Ability to create executive-ready narratives that explain quality trends, model risks, release readiness, and recommended actions.
Leadership and Communication Skills
Lead evaluation planning, quality governance, stakeholder reviews, and cross-functional improvement discussions.
Mentor team members on evaluation methodology, simulation design, AI behavior analysis, and insight-driven reporting.
Communicate complex AI quality findings in a clear, structured, and action-oriented manner.
Operate with strong ownership, attention to detail, curiosity, and a continuous improvement mindset.
Preferred Tools and Platforms
AI evaluation, simulation, and conversation quality automation platforms
Transcript analytics, speech analytics, and contact center intelligence tools
LLM evaluation frameworks, prompt evaluation approaches, and model observability solutions
Dashboarding and analytics tools such as Power BI, Tableau, Looker, Excel, SQL, Python, or equivalent
Conversational AI platforms, voice AI platforms, agent assist solutions, and contact center technology ecosystems
Success Measures
Improved accuracy, groundedness, reliability, and consistency of Conversational AI and LLM-powered experiences
Reduction in hallucination risk, fallback rates, repeat contacts, avoidable escalations, and unresolved customer journeys
Improved containment, task completion, intent performance, response relevance, and customer experience outcomes
Clear visibility into model quality trends, release readiness, defect themes, and improvement priorities
Actionable insights adopted by product, design, engineering, and client teams to improve AI performance over time
Required
Preferred