EVA: A New End-to-end Framework for Evaluating Voice Agents
ServiceNow
·
May 12, 2026
·
video
Voice agents are increasingly being deployed in customer service and conversational AI applications. However, existing evaluation frameworks assess text-based dialogue or evaluate acoustic and linguistic components in isolation, failing to capture emergent behaviors and failure modes of end-to-end voice systems under realistic conditions. Tara Bogavelli and Gabrielle Gauthier Melançon are presenting EVA, a framework that evaluates voice agents end-to-end across two dimensions: accuracy and experience. Accuracy describes whether the voice agent is able to correctly complete the user's request. Experience describes the quality of the conversational experience from the user side. We release EVA with an initial airline domain comprised of 50 tasks. We evaluate 20+ systems — cascade as well as audio native — and find that a fundamental tradeoff exists between accuracy and experience. No system performs well on accuracy and experience simultaneously, indicating that the development of voice agents that excel across both dimensions remains an open problem.
https://www.youtube.com/watch?v=awnHL7AARZM