The role
Job description
Job Details:
Position Title : AI & Optimization Lead
Experience : 12+Years
Location : Noida/Gurgaon/Pune/Bangalore(Hybrid)
Shift : 1:30PM- 11:30PM (UK Shift)
Function: Customer Experience / Conversational AI
Role Type: Client-facing , AI Optimization & Performance Leadership
Role Overview
We are seeking an experienced techno-functional leader to drive the optimization, performance improvement, and continuous evolution of deployed Google Voice AI and Agentic AI solutions for a leading US Investment Management client.
The role requires a combination of expertise in Google’s Conversational AI ecosystem, Agentic AI concepts, US BFSI domain knowledge, and investment management business processes. The individual will be responsible for evaluating production AI behavior, identifying hallucinations and other agent failure patterns, improving response quality and execution reliability, and translating analytical insights into measurable business outcomes.
The candidate will act as a trusted partner to client stakeholders, bridging business, operations, and technology teams to ensure that deployed AI solutions are accurate, reliable, responsive, secure, and aligned with business objectives.
Key Responsibilities
1. Voice AI & Conversational AI Optimization
Lead the continuous optimization of deployed Google Voice AI solutions, improving self-service completion, containment, intent recognition, conversation quality, and customer experience.
Analyze call recordings, transcripts, conversation flows, fallback patterns, transfers, and customer drop-offs to identify improvement opportunities.
Optimize intents, entities, prompts, conversation design, fulfillment logic, integrations, and human-agent handoffs.
Investigate production issues involving conversational flows, APIs, backend integrations, telephony, and response generation.
Drive improvements in first-contact resolution, average handling time, customer satisfaction, and operational efficiency.
2. Agentic AI Quality, Reliability & Optimization
Evaluate deployed AI agents for accuracy, consistency, groundedness, instruction adherence, and successful task completion.
Identify and investigate hallucinations, fabricated facts, unsupported responses, incorrect summaries, inconsistent answers, and responses that conflict with approved business rules or source data.
Analyze failures arising from prompt design, model behavior, context management, retrieval quality, tool selection, orchestration, and backend integrations.
Evaluate multi-step agent behavior, including planning, reasoning outcomes, tool calls, API execution, retries, recovery, and escalation to human agents.
Identify instances where agents invoke incorrect tools, pass inaccurate parameters, fail to complete tasks, repeat actions, or provide misleading confirmation of successful transactions.
Establish and refine evaluation criteria, test scenarios, quality scorecards, and acceptance thresholds for conversational and agentic use cases.
Partner with AI engineering and platform teams to implement corrective actions and validate improvements through regression testing and production monitoring.
Assess opportunities to improve agent autonomy while maintaining appropriate guardrails, business controls, and human oversight.
3. Latency, Performance & Production Observability
Monitor and investigate slow responses, delayed turn-taking, excessive response times, and end-to-end conversation latency.
Analyze latency across speech recognition, model inference, agent orchestration, tool execution, API calls, text-to-speech, and telephony components.
Identify bottlenecks causing long pauses, interruptions, timeouts, repeated responses, and degraded voice experiences.
Correlate technical performance metrics with customer behavior, abandonment, transfers, and conversation completion.
Work with platform and engineering teams to establish service-level indicators, performance thresholds, alerting, and root-cause analysis processes.
Distinguish between model-quality issues, infrastructure or integration latency, and conversational-design problems.
Drive optimization initiatives to improve responsiveness, reliability, availability, and scalability.
4. AI Evaluation, Testing & Continuous Improvement
Develop and maintain evaluation frameworks covering factual accuracy, hallucinations, relevance, instruction following, tool-use correctness, API success, task completion, and response latency.
Build and maintain representative test scenarios using historical interactions, business scenarios, edge cases, and failure patterns.
Analyze evaluation results to identify recurring defects, regression trends, and areas requiring prompt, model, workflow, or integration changes.
Establish quality baselines and track improvements across releases using measurable metrics.
Partner with QA and engineering teams to integrate AI evaluations into functional testing, regression testing, release validation, and production monitoring.
Use automated testing, conversation replay, trace analysis, and observability tools to improve the speed and effectiveness of optimization.
Ensure that improvements are validated against realistic customer journeys and applicable business requirements before production deployment.
5. US Investment Management & BFSI Domain Expertise
Understand investment servicing workflows, customer journeys, contact center operations, and relevant business terminology.
Work with client stakeholders to optimize use cases such as account and portfolio inquiries, balances and holdings, transaction status, distributions, statements, tax documents, and account maintenance, as applicable.
Ensure AI responses align with approved business content, source-system data, authentication requirements, privacy expectations, and relevant regulatory controls.
Identify business-process gaps and opportunities to automate repetitive customer interactions while preserving appropriate safeguards.
Translate investment management business requirements into conversational enhancements, agent capabilities, evaluation scenarios, and measurable business outcomes.
6. Analytics, Insights & Business Outcomes
Analyze interaction data, agent traces, evaluation results, API logs, operational dashboards, and quality reports to identify trends and underlying causes.
Develop dashboards and performance reports covering containment, hallucination rates, response accuracy, latency, tool-call success, API failures, task completion, and customer experience.
Segment defects by severity, root cause, business impact, and recurrence to prioritize corrective actions.
Establish optimization roadmaps based on customer impact, operational efficiency, technical feasibility, and business value.
Present findings, recommendations, risks, and realized benefits to client stakeholders and senior leadership.
7. Client Leadership
Serve as the primary techno-functional point of contact for client discussions related to AI solution performance, quality, and optimization.
Lead performance reviews, optimization workshops, root-cause discussions, and governance meetings.
Convert business feedback and technical findings into a prioritized improvement backlog with clear ownership, timelines, and success criteria.
Coordinate with architecture, engineering, QA, operations, product, and platform teams to drive issue resolution and enhancement delivery.
Communicate complex AI concepts and performance issues clearly to both technical and non-technical stakeholders.
Establish structured governance for production changes, release readiness, risk management, and continuous improvement.
Required Qualifications & Experience
10–14 years of overall experience in technology delivery, Conversational AI, contact center transformation, or enterprise AI solutions.
Strong techno-functional knowledge of Google Conversational AI platforms, preferably Dialogflow CX and/or Google Cloud conversational agent platforms.
Experience managing or optimizing deployed Voice AI solutions in production environments.
Working knowledge of Agentic AI, LLM-based agents, tool calling, orchestration, and multi-step workflows.
Demonstrated ability to identify and investigate hallucinations, incorrect responses, agent execution failures, and performance bottlenecks.
Understanding of AI evaluation methods, automated testing, quality measurement, and production observability.
Good understanding of US BFSI, with direct exposure to Investment Management, asset management, wealth management, brokerage, or investment servicing preferred.
Strong analytical, problem-solving, and root-cause analysis capabilities.
Excellent client management, communication, presentation, and stakeholder engagement skills.
Experience coordinating cross-functional teams and driving measurable improvements in live customer-facing solutions.
Preferred Technical Skills
Google Dialogflow CX and Google Cloud conversational AI capabilities.
Gemini-based agents and relevant Google agent development and orchestration technologies.
LLM evaluation, hallucination detection, grounding, prompt optimization, and response-quality assessment.
Agent traces, tool-call analysis, API and webhook troubleshooting, and workflow observability.
REST APIs, JSON, authentication, enterprise integrations, and backend system connectivity.
Voice architecture, speech-to-text, text-to-speech, SIP/telephony, and contact center integrations.
SQL, Excel, BI dashboards, and analytical reporting.
Familiarity with Genesys Cloud, Amazon Connect, or other CCaaS platforms.
Exposure to automated AI evaluation and testing frameworks, including custom evaluation pipelines or relevant industry tools.
Key Performance Indicators
Reduction in hallucinations, unsupported responses, and incorrect agent actions.
Improvement in response accuracy, groundedness, and adherence to approved business rules.
Reduction in average and high-percentile response latency.
Improvement in tool-call success, API reliability, and end-to-end task completion.
Improvement in Voice AI containment, self-service completion, and customer experience.
Reduction in recurring production defects, avoidable escalations, and failed conversations.
Timely delivery of optimization initiatives and measurable business benefits.
Client satisfaction and confidence in the reliability of deployed AI solutions.
Critical Competencies
Strong analytical mindset with the ability to move from symptoms to root cause.
Ability to connect AI behavior and technical performance to business outcomes.
Strong customer relationship and stakeholder management skills.
Structured problem-solving and data-driven decision-making.
Ownership, accountability, and a continuous improvement orientation.
Ability to influence engineering and business teams without necessarily owning every technical component.