ServiceNow Virtual Agent powered by Now Assist can significantly enhance self-service experiences, so it is equally important to evaluate the quality of those interactions to ensure users receive accurate, effective, and reliable support. Organizations typically use several methods to assess Virtual Agent performance.

• Traditional Reviews: Manual conversation reviews, surveys, and spot checks that provide insights but are time-consuming and difficult to scale.
• Conversational Insights: ServiceNow Conversation insights application provides inferred CSAT and metrics such as resolution, empathy, and effort to measure user satisfaction. 
• Now Assist Evaluation Skills: Automates conversation quality assessment, provides reasoning behind scores, and supports continuous improvement of Virtual Agent performance.

Now Assist Evaluation Skills and Conversation Insights application together provides a complete picture of virtual agent effectiveness, from system performance to end-user satisfaction. While Now Assist evaluation skills help in measuring Virtual Agent design and model performance whereas Conversation insights app focuses on measuring customer satisfaction and effort. This article focuses on Now Assist Evaluation skills activation, configuration and usage details. 

 

 

Now Assist Evaluation Skills Activation and details

The Now Assist Evaluation Skills can be enabled from the Now Assist Admin Console under Platform Skills. There are eight evaluation skills , each designed to assess a specific aspect of Virtual Agent conversation quality.

 

• Intent Accuracy – Verifies whether the Virtual Agent correctly understood the user's request.
• Slot Filling – Evaluates how accurately required information is captured.
• Conversation Flow – Identifies repetitive loops or conversations that fail to progress.
• Context Retention – Measures the agent's ability to remember earlier user inputs.
• Coherence – Ensures responses are logical and well-structured.
• Truthfulness & Hallucination Detection – Validates that responses are grounded in available knowledge and context.
• Conciseness – Checks whether responses are clear and avoid unnecessary detail.
• Topic Classification – Categorizes conversations for reporting and trend analysis.

 

Steps involved in Conversations Evaluation

Not every Virtual Agent conversation is evaluated. By default, ServiceNow evaluates a 10% sample of daily conversations , with a limit of 200 conversations per day. This limit can be increased/decreased through a system property if needed (sn_na_conv_eval.maxEvaluateCount). Since each evaluation skill consumes 1 Assist and all eight skills may be executed for a conversation, sampling helps optimize Assist consumption while still providing meaningful insights.

 

Additionally, some conversations are excluded from evaluation, including Conversations involving restricted Knowledge Base articles, HR-related conversations and Very short conversations.

A transcript is generated from each eligible conversation and sent to the LLM for evaluation. Once processing is complete, the results are stored in the Evaluation and Metrics tables for analysis and reporting.

 

 

Evaluation Methods

We can choose one of the following evaluation approaches:

• Scheduled Evaluation: A daily scheduled job evaluates conversations completed during the day. Job name: Execute Evaluations
• Real-Time Evaluation: A Flow triggers the evaluation immediately after a conversation ends. 
• Historical Evaluation: A one-time batch process can be used to evaluate past conversations up to 100 conversations by adding a conversation records query (sys_cs_conversation) to the "sn_na_conv_eval_evaluation_set" table. 

Note: Scheduled and real-time evaluation methods are mutually exclusive. Only one approach should be used at a time.

 

Where to view the results of Evaluation

Evaluation results can be viewed in the AI Control Tower. The dashboard provides a score ranging from 0 to 5 for each evaluation metric, along with the total number of conversations evaluated and a breakdown of conversations by size (small, medium, and large).

For every score, the LLM generates constructive reasoning to explain the evaluation. Additionally, human reviewers can provide their own scores and comments to validate or challenge the LLM's assessment, helping to further refine and improve the evaluation model over time.

➡️ To conclude, Now Assist Evaluation Skills act like a quality check for Virtual Agent conversations, helping organizations deliver better, more accurate, and more helpful support experiences.