The role
Job description
Location: Noida, Gurugram, Pune and Bengaluru- Hybrid
Band: B / C
Total Experience: 2-6yrs
Function: Customer Experience / Conversational AI
Role Type: Conversational AI Analysis & Reporting (reports to Delivery Lead, Band D2)
Role Overview
We are seeking an experienced Conversational AI Analyst to support the optimization and performance improvement of deployed Google Voice AI and Agentic AI solutions for a leading US Investment Management client. The role sits within the AI & Optimization delivery team and reports to the Delivery Lead (Band D2).
The analyst will be the team's primary hands-on analysis and reporting resource: reviewing production conversations, agent traces and operational data to identify the real gaps in AI behavior, such as hallucinations, intent and flow failures, incorrect tool use, latency issues and business-rule violations, and translating them into clear, evidence-backed findings and recommendations. Strong understanding of the US investment management domain is essential to distinguish genuine solution gaps from expected customer or process behavior.
The role is largely analytical and delivery-focused, with limited direct client interaction. Findings, reports and recommendations are reviewed with and presented through the Delivery Lead.
Key Responsibilities
1. Conversation & Voice AI Analysis
Analyze call recordings, transcripts, conversation flows, fallbacks, transfers and drop-offs for deployed Google Voice AI solutions to identify improvement opportunities.
Perform root-cause analysis to separate intent and entity recognition issues, prompt and conversation design gaps, backend or integration failures and telephony or speech-recognition problems.
Identify the actual gaps behind poor containment, low self-service completion and avoidable transfers, rather than reporting symptoms only.
Recommend improvements to intents, entities, prompts, conversation design, fulfillment logic and human-agent handoffs, and support their validation after release.
2. Agentic AI Quality & Failure Analysis
Review agent conversations and traces to assess accuracy, groundedness, consistency, instruction adherence and task completion.
Identify and document hallucinations, fabricated or unsupported responses, incorrect summaries and responses that conflict with approved business rules or source data.
Analyze multi-step agent behavior, including tool selection, parameters passed, API execution, retries, recovery and escalation, and flag incorrect tool calls, repeated actions and misleading transaction confirmations.
Classify defects by type, severity, root cause, business impact and recurrence, and maintain the defect log and failure-pattern library.
Provide clear evidence (transcripts, traces, logs) to support AI engineering and platform teams in fixing issues.
3. Latency & Performance Analysis
Monitor and analyze response times, turn-taking delays and end-to-end conversation latency across speech recognition, model inference, orchestration, tool and API calls, text-to-speech and telephony.
Identify bottlenecks behind long pauses, interruptions, timeouts and repeated responses.
Correlate performance metrics with abandonment, transfers and conversation completion.
4. Evaluation & Testing Support
Build and maintain test scenarios from historical interactions, business scenarios, edge cases and known failure patterns.
Execute and analyze evaluation runs covering factual accuracy, hallucinations, relevance, instruction following, tool-use correctness, API success, task completion and latency.
Track quality baselines and regression trends across releases, and validate fixes against realistic customer journeys before production deployment.
Support QA and engineering teams with conversation replay, trace analysis and production monitoring.
5. US Investment Management & BFSI Domain Application
Apply knowledge of investment servicing workflows, contact center operations and customer journeys to assess whether AI responses and actions are correct and complete.
Review use cases such as account and portfolio inquiries, balances and holdings, transaction status, distributions, statements, tax documents and account maintenance.
Check that responses align with approved business content, source-system data, authentication requirements, privacy expectations and applicable regulatory controls.
Identify business-process gaps and candidate use cases for automation or enhancement, and document them as clear requirements and evaluation scenarios.
6. Reporting, Insights & Dashboards
Develop and maintain dashboards and recurring reports covering containment, hallucination rate, response accuracy, latency, tool-call success, API failures, task completion and customer experience.
Analyze interaction data, agent traces, evaluation results and API logs to surface trends, anomalies and underlying causes.
Prepare analysis decks, weekly and monthly performance summaries and optimization recommendations for the Delivery Lead, including prioritization inputs such as impact, effort and recurrence.
Track the status and measured benefits of closed improvement items.
7. Collaboration & Delivery Support
Work closely with the Delivery Lead to convert findings into a prioritized improvement backlog with clear owners and success criteria.
Coordinate with AI engineering, QA, operations and platform teams to clarify issues and confirm fixes.
Participate in client working sessions and review meetings as a subject-matter contributor, and present analysis when required.
Communicate findings clearly, with data and examples, to technical and non-technical audiences.
Required Qualifications & Experience
4–5 years of experience as a Conversational AI Analyst, Voice AI/Chatbot Analyst, or in conversation analytics, contact center analytics or AI quality analysis, with exposure to production environments.
Hands-on experience analyzing live conversations, transcripts and logs to identify real defects and improvement opportunities.
Working knowledge of Conversational AI platforms, preferably Google Dialogflow CX and/or Google Cloud conversational agents.
Working knowledge of LLM-based and Agentic AI concepts, including prompts, grounding, tool calling, orchestration and multi-step workflows.
Demonstrated ability to identify hallucinations, incorrect responses, tool or API execution failures and performance bottlenecks, and to trace them to root cause.
Good understanding of US BFSI with direct domain exposure to Investment Management, asset management, wealth management, brokerage or investment servicing.
Strong analytical skills, with proficiency in Excel and SQL, and experience building dashboards or reports in a BI tool.
Clear written and verbal communication, with the ability to present findings concisely.
Bachelor's degree in Engineering, Computer Science, Analytics, Business or a related field.
Preferred Technical Skills
Google Dialogflow CX, Gemini-based agents and Google Cloud conversational AI capabilities.
LLM evaluation, hallucination detection, grounding and response-quality assessment.
Agent trace and tool-call analysis, and API and webhook troubleshooting using REST APIs and JSON.
Speech-to-text, text-to-speech and telephony fundamentals; familiarity with CCaaS platforms such as Genesys Cloud or Amazon Connect.
SQL, Excel, Power BI, Tableau or similar BI tools; basic Python for data analysis is a plus.
Exposure to automated AI evaluation and testing frameworks or custom evaluation pipelines.
Key Performance Indicators
Accuracy, depth and timeliness of gap and root-cause analysis.
Share of identified issues validated and accepted as genuine gaps by the Delivery Lead and engineering teams.
Quality and timeliness of dashboards and recurring performance reports.
Coverage and quality of test scenarios and evaluation runs.
Measurable improvement in hallucination rate, response accuracy, latency, tool-call success and containment tied to recommendations.
Reduction in recurring defects and repeat failure patterns.
Stakeholder feedback on clarity and usefulness of insights.
Critical Competencies
Analytical mindset with the ability to move from symptoms to root cause.
Domain depth in investment management to separate real gaps from expected behavior.
Attention to detail and evidence-based reporting.
Structured problem-solving and data-driven recommendations.
Ownership, accountability and a continuous improvement orientation.
Collaboration with engineering, QA and business teams, and readiness to grow into broader ownership.