AI Agent Eval Studio is a ServiceNow application for systematically testing, evaluating, and improving Now Assist AI agents and use cases. It automatically builds evaluation context from agent definitions, tools, documentation, and execution history; generates meaningful test cases; supports human review; executes approved evaluations; and scores results using deterministic validators and AI-powered judging.

The app gives teams a repeatable way to measure agent quality, identify regressions, validate business outcomes, and continuously improve AI experiences before they reach production.

- Context-aware evaluation generation Builds evaluation context from agent definitions, instructions, tools, documentation, and historical execution data.
- Automated test-case generation Generates business-level, agent-level, and tool-level evaluation suites tailored to the selected AI agent or use case.
- Sufficiency and clarification workflow Detects missing context and guides users through targeted clarification questions before generating evaluations.
- Human review and approval Review, edit, approve, or reject generated test cases before execution.
- Validator-based testing Supports deterministic checks for expected outcomes, tool usage, data changes, response quality, and multi-step behavior.
- AI-powered judging Uses an LLM judge to evaluate nuanced responses and business outcomes that cannot be validated through deterministic rules alone.
- Real-agent execution Runs approved evaluation cases against the configured ServiceNow AI agent or use case.
- Execution tracking and scoring Tracks evaluation progress, captures results, and combines deterministic and AI-judged scores.
- Regression detection Compares evaluation results across runs to identify performance changes and recurring failures.
- Comprehensive reporting Provides pass rates, failed cases, regression insights, validator results, and executive summaries.
- Reusable validator authoring Enables teams to create, review, and reuse custom validators for organization-specific behaviors.
- ServiceNow-native integration Built on ServiceNow scoped application architecture and integrated with Now Assist, `sn_aia`, and GenAI capabilities.