Staff Systems Application Developer - AI Agent Analytics

Job Highlights

• 3–6 years in analytics with AI/ML system evaluation experience, expertise in AI agent QA and prompt engineering
• Track and analyze AI agent performance metrics, identify failure patterns, define AI productivity KPIs, maintain real-time dashboards, run QA cycles, and prepare leadership briefings

About Saviynt

Saviynt's AI-powered identity platform manages and governs human and non-human access to all of an organization's applications, data, and business processes. Customers trust Saviynt to safeguard their digital assets, drive operational efficiency, and reduce compliance costs. Built for the AI age, Saviynt is today helping organizations safely accelerate their deployment and usage of AI. Saviynt is recognized as the leader in identity security, with solutions that protect and empower the world's leading brands, Fortune 500 companies and government institutions.

AI agents are only as good as the intelligence behind them. At Saviynt, our customer support runs on autonomous AI agents that triage, route, and resolve cases without human intervention and we're just getting started. The next frontier is measurable: higher deflection, faster closure, and agents that learn from every interaction. This role sits at the center of that mission. As our Business Analyst - Customer Support (AI), you are the analytical engine that tells us exactly where our agents are winning, where they're failing, and what it will take to push autonomous resolution rates higher.

WHAT YOU WILL BE DOING

1. Performance Analytics

Own the performance analytics for our AI agent fleet — tracking deflection rates, autonomous closure rates, escalation triggers, confidence score distributions, and CSAT by resolution type to give us a precise, real-time picture of where agents are succeeding and where they're not.

2. Escalation Analysis

Dig into cases that escape AI resolution — identifying the intent clusters, knowledge gaps, and routing failures that cause unnecessary escalations, and translating those findings into specific improvements for agent training and prompt refinement.

3. KPI Framework

Define and govern the KPI framework for AI productivity — including human-time-saved per resolved case, agent-assisted vs. fully autonomous closure ratios, and the quality metrics that validate whether AI resolution is genuinely satisfying customers or just closing tickets.

4. Real-time Dashboards

Build and maintain real-time dashboards that surface AI agent health, case closure velocity, and deflection trends — giving the support leadership a live signal on how the AI operation is performing at any given moment.

5. QA & Evaluation

Run structured QA and eval cycles on AI agent outputs — reviewing response accuracy, tone calibration, and resolution completeness to catch quality drift before it shows up in CSAT or re-open rates.

6. Leadership Briefings

Prepare concise, evidence-based briefings for leadership on AI agent performance and productivity gains — translating what the data shows into clear narratives about where to invest next to push resolution rates higher.

WHAT YOU BRING

• 3–6 years of experience in analytics or business intelligence, with at least some of that time spent analyzing AI, ML, or LLM powered systems in a production environment — you've measured how models behave in the real world, not just in test sets.
• Hands on experience with AI agent evaluation — you know how to design rubrics, run QA cycles, and define what "good resolution" actually means in a support context. You can calibrate quality, not just count tickets.
• Sharp analytical instincts — you can tell the difference between a deflection rate that's improving because agents are getting better and one that's improving because frustrated users are giving up. You ask the next question, not just the first one.
• Clear communication — you can turn a complex AI performance analysis into a crisp recommendation that a support leader can act on immediately, without needing a technical deep-dive to understand the point.
• Familiarity with prompt engineering or context engineering enough to have informed opinion on why an agent responded the way it did and what a better prompt structure would look like.

Why Join Saviynt?

• ✅ AI-powered identity security leader
• ✅ Work with cutting-edge AI agent technology
• ✅ High-growth Platform as a Service company
• ✅ Innovative and dynamic work environment
• ✅ Opportunity to shape the future of AI support

Verified by Employee Table — free to apply, no registration fee required.