all right let's go ahead and get started thank you all for joining good morning or good afternoon wherever you are in the world welcome to the virtual agent academy as always please be sure to visit our virtual agent academy community at community.servicenow.com there you'll see you'll be able to explore best practices and update content you'll see uh articles and blogs written by product managers and subject matter experts use the experience of others ask questions give answers and learn all the latest we have about our product if you're watching us on youtube click the subscribe button to see more content such as this as well as all our previous virtual agent academy recordings that's also those are also available on our community site if you like what you're seeing click the like button if you have any questions feel free to comment what i also like to share is that we also have other products that do academies our mobile academy is every alternating every other alternating tuesday morning slash afternoon and we have an hr academy that is on every fourth tuesday of the month and let's welcome to the welcome everyone to the main event this in this webinar we're going to gain insights into your virtual agent with conversational analytics so our guest speakers today are yuko rocky who is a product manager for conversational interfaces returning champion gaurav goyal who's a software engineer with the team and myself victor chen so just a brief overview of what we're going to cover today we're going to do an overview of the conversational analytics or what it is and when it's available and then yuko and gaurav are going to do an exercise as to give us a tour of conversational analytics and how to explore all your analytics metrics edit them uh filter them etc and then finally we will do q a throughout the uh webinar and throughout the exercise feel free to answer uh to ask key questions in the q a channel and uh i and others on the call service now folks on the call will monitor it so overview here for conversational analytics we're going to enable deep insights into your conversational data with minimum performance overhead on your customer interfaces you may be familiar with conversational analytics in our parish release that used pa or performance analytics well in the new quebec release we're enabling a whole new um ux visualization for conversational uh on analytics this uh is using a new technology that work that we insert now internally called seismic it's a brand new ui uh built on ui builder so not only are you able to uh see and access all these metrics that we're gonna go over in a moment you will be able to build your own edit them modify them and modify the formula modify uh the ui whether you want to add more metrics or take some out that you feel may not be relevant or important to you this computation analytics dashboard fulfills the persona of whether you're an executive that looks wants to see the performance of your topic conversations and authors who wants to see the performance of a particular topic or particular nlu or a center of excellence just wanted to make sure that everything going on in your conversation in your conversations and in your organization is in tip-top shape while users self-serve and self-solve on using your virtual agent whether it's our out-of-box data or our filter data you will most likely be able to find the questions you need in an interactive and easy to use fashion some example analytics we have you can kind of see some in the screenshot we have here but don't worry if it's unclear we're going to go we're going to do a tour just a bit but you can see analytics such as uh usage so active users conversations etc duration of these conversations we don't want conversation to be too long so you can track duration of folks uh spending time in virtual agent the self-solver rate that's very critical for uh our customers right our ver is virtually agent helping our employees answer their questions without having to go to a live agent or without having a bad experience we want good experiences and solutions for our customers feedback right after conversations there are surveys sometimes are those surveys negative or positive nlus i mentioned and a lot more so conversational analytics again in the quebec release is available as a store app so if you visit our store at servicenow store.servicenow.com you can uh install conversational analytics and what's great about this being on a store release as opposed to a biannual family release paris quebec rome etc is that you can expect more uh more constant updates than the twice a year update so we may provide more ui uh more uh experiences or bug fixes what have you they will be uh more frequent than than just twice a year great so without further ado i'm now going to hand it over to yuko and gaurav to show you a conversation analytics you go girl take it away okay thank you so much victor let me share my screen um sorry about that okay so as victor mentioned and this new conversation i have like dashboard is not called virtual agent dashboard just to be sure it's called conversational um analytics because it is inclusive of metrics of virtual agent and live agent and we plan to introduce more and more live agent operational side metrics in the future how you access this um you type in conversational analytics obviously after you install your store app and right underneath here under collaboration section you will get the virtual agent dashboard once the the dashboard loads um as victor mentioned the top overview page will see executive targeting metrics business metrics uh that you can you can take a look at how is my instance doing in a nutshell um as you can see there is a tab structure uh for this dashboard over the usage conversations users nlu prediction and custom events and i will go through each one of them very quickly first a quick look how is active users doing how the conversation is doing what percentage of all of my conversations are at least completing to the end of the topic node and have the positive um positive feedback and what percentage of topics are completing to the topic and these are the top metrics that we are creating out of the box for the executive persona now here is the calendar picker in most of the cases you can choose no last week next week or you can choose custom date range as you wish for our example let's choose the last three months this will dynamically change the data in the dashboard as you can see if you want to dig further into the active user's usage you can see the trend line but you can also click into it and see exactly how the the active users increase first thing happening you want to be able to see them on a daily basis or a monthly basis there are multiple options for you to see the chart the drilling down into the detail information is very critical for most customers so really focused on this feature download will allow you to download the excel sheet in for this time line duration now the performance section here's a category um in quebec we introduce the concept of associating a topic to a category so imagine if you are um an hr is the [Music] center of excellence manager um departmental manager or if you are an itsm self-service manager the interest that you have in how my graduation deployment is completely different so i've created the view uh depending on who you are uh so that this one is if you happen to really only care about itsm how are my conversations doing are they all um ending well um is everyone going to the live agent transfer how are the topics being used what's the diff uh distribution of the channel and these are the kind of things that you can see this only for the conversation that hit the topic that belong in your category so let's go back to the overview page just really quickly here is the very quick overview of how each of your topics are doing i hear the distribution of user feedback the channel distribution and in quebec we are using the interaction table in the column name state and another column name called state reason to track how exactly uh conversation has ended so some conversation ended because agent closed the live agent or some conversation age close because user closed to live asian before the live agent even picked up the phone uh pick up the chat so those are the the very detailed granular uh distribution of how the conversation is ending if you think this is too much don't worry we created a method to group them in the way how you would like to do so in general this this dashboard is editable not only on the ui side but also on a data modification side um before i go further into different tabs uh let me stop here and if you see um if there's any questions from the audience yeah you go um one question of the audience from david how do you determine active users active users are the users who have had at least one conversation during this period that you have picked from your calendar picker got it and then another question uh is from mary will you cover the formula being used to determine the self-solve rate yes i will i will show you that um okay um so here was the the overview page but let's just imagine that you want to dig further into how your virtual agent you know deployment is doing specifically what percentage of your conversation is escalating to live agent or maybe um depending on who you your customer set up is maybe the chatbot goes directly to the live agent so here's a breakdown you can see the language breakdown the duration um and in quebec we are introducing actionable notification so this is a very interesting metrics after you send an actionable multiplication and end user is given the notification some of them click or some of them completely ignore so you want to be able to see what percentage of your actionable notification has been engaged and completed a topic issue of the resolution is another flagship feature we um shipped in quebec and what this is is that if an end user filed a ticket before a live agent picks up that ticket to handle the issue if there is already an existing topic that matches the issue that the end user files that we will send actionable notifications and then bot will would try to solve the problem for you so what this means is that you know okay there is already a topic that says vpn connectivity and the end user filed an issue of vpn connectivity i want uh i have some intuitive activity on a portal yesterday they just say that the system will pick up that hey there's already a topic a bot that can probably solve the issue for this end user will send the issue of the resolution actionable notification um and out of which 34 of them 94 has actually accepted um and completed an issue so that's what they showed the resolution metrics with me i need to pick up my speed here another thing i want to be able to show is a very detailed drill down of how each of the conversations are doing or maybe before going here if i happen to be an author persona so we saw an overview which is for the executive persona um a center of excellence who happens to be an idea send a category persona right account of the specific category view we also looked into a little bit of a drill down um you know subject matter expert type which happened to be in a usage issue at the resolution but let's look at the author persona so if your customer happens to be an other persona i want to be able to fix any topics that may have an issue we have created out of box quite many widgets here um the most interesting one is the worst performing topics uh unfortunately in our industry we just want to know what's going wrong instead of what's going right we won't be able to see huh interesting to change my password seems to be always incomplete what does incomplete mean it means that the end user did not finish to the last node of this topic change my password that's not really good i will keep an eye on that transfer to live agent oh that's interesting whenever this what's up card topic is running it seems like the end user tends to transfer to live agent which means that this topic is definitely not being useful so those are what can give you the insight into each of these here spokes use um another new quebec features we have created a direct integration to integration hub from the virtual agent designer and so if you want to see what are the types of actions and subflows being used from each of these integration hub spokes so let's just choose microsoft azure ad spokes and i can see that these three actions from the azure 80s books have been used and all of these are used by the parent topic at distribution list so that's what the spokes metrics are showing i'd like to go back to the topic issues so earlier we said something seems to be wrong with change my password and the what's up topic isn't doing well okay well let's go further into what does this actually mean here's a list of topics table and you can see you know what percentage of how they have been initiated seems like show notification is running the most and then change my password oh that's a really bad news because change by password is not always incomplete and yet it's always being initiated meaning customers and users like to invoke this topic let's go further into the detail of this topic what's going on here as you can see here's a trend of the usage itself as usual you can change the view daily view versus monthly new and as we saw earlier incomplete percentage is quite high 48 although not nobody is uh is going into the live agent and here is where we can see as an author persona maybe there is something that is causing the end user to drop out in the middle of the topic now forgive me this is all the uh you know the mock data um so um the actual node and and whatnot might not make sense but based on this information you can see that whenever you are we are showing them try again or asking them to yet again you know make a decision a customer seems to be dropping off so the last visiting node is a quite interesting um information we can provide to the author persona let me pause here and collect some questions yeah you go we have a question let me go back a bit uh to katie is there a way to see if a user opened a specific kb article from virtual agent if that article helped the user fix the problem yes um great question so i'm currently working on a um the next release for august which is going to make it even easier however even in this release that is available as of february we can do that how do we do so as an author you know which topic is actually the one that is showing um the kb article kb article can be shown to the customers either by having a contextual topic a block inside a particular topic or kb article is being shown because of ai search topic block so we have created a special section where there is a topic block usage so in this case ai search topic block usage or contextual search topic block usage which will give you the distribution and execution count of what topics and how many times that search has been shown to the end user additionally if you're interested in the number of users that hit a particular search results you go to the users list okay and from here you will see i want to see the number of users that had the conversation and from here you just say you know i know that there is a um topic and there is a topic that is called let's just say printer issues which i have included a contextual search topic block in it and and whenever you hit this printer issues i know that end users are hitting the search topic block so this is the way to see how many results how many users unique users have actually had the conversation that happened to have used this particular topic and had this you know completion stages to be completed but furthermore if you know specific node happened to have had a search topic block what you can do is go here copy node um so what i'm doing here is that i'm looking at all conversation all users list this is all the users list from the calendar picker that we talked about earlier um that happened to have hit a topic called printer issues which had the topic node and the topic node name happened to be a set search query okay with this i know how many users actually were served with regard to the printer issues and saw the search query um yeah would be the list and i'm sorry i'm i'm going this on the ad hoc basis based on what you ask me on a question so maybe it's not giving you enough users but um this is a level of filtering that you can do okay i answer your question any other questions uh we have let's leave questions at the end uh given our time constraints let's uh go into some of the ways you can uh create custom events or perhaps modify a metrics and some something along that angle yeah absolutely um yeah so that's just before going there um there is also a nlu tab just want to show you that here's an nu prediction uh trend and you can change the weekly daily and here's the list of the top topics in order of usage that match the prediction if you are nlu enabled on a per topic level um but what okay so what um victor was saying is that as i said earlier this dashboard is not only configurable from a ui perspective but also from a data perspective so let me show you how you change the ui so let me let's just say that um you know on the overview page i want to add another scorecard here the four isn't enough i want like another one here and that the way how you do that is you you go here and you click on edit page and this is a new ib builder page uh completely built on a new um ui framework and what you would do is you edit an original scope and you click over here and you add um data visualization um you can add another let's just say another widget here you add a data source you click on a table yeah so you get the idea this is exactly how you would um pick uh i don't know let's just click anything um either do you want the single score you want the lines um horizontal bar so this is how you would add your own widget so that if you want to make the changes and add other widgets that comes the way the data comes from the glide table um change and manipulate the field and look of this dashboard that was with regard to the ui builder now let me talk about how you want to add your very specific um data including the previous question that i got which is how do i change the formula of the self salt self salt now how do you do that is first let's talk about specific data not changing the existing formula of a specific data although i have given you so many out of the box widgets which is even configurable through the filtering maybe you still want something that is very special to you and how do you do that we created a mechanism called custom events and custom events will always come out like this and i'll just show you how you build this earlier i typed in conversational analytics and show the graduation dashboard immediately underneath it with the exact same store app you will get the event configuration and formula override modules to go to the event configurations this is where you would be creating a brand new event that would be sent from the virtual agent server to the dashboard backend the national board backend will understand whenever such event occurs and tries to create auto-create a pie chart or the line chart on behalf of you so i created this earlier let's take a look at this some customers gave me a feedback and that this customer is always tracking what their executives experiences would be with the virtual agent because whenever uh the the cto or cio experiences the internal virtual agent and it's an issue they get a call and they don't they want to avoid that scary call from their executive so they want to be always monitoring what's going on with the vip usage so here's the thing you know you create a custom event um you know source table from a conversation because you're trying to track what's happening with the vip conversation and all the condition is that from consumer you know consumer fields and we happens to be a vip is true and um let's say properties of maybe we will say how is this doing with regard to i don't know um conversation type submit and when you submit what this will happen and by the way this is an event so it will only go for the moving forward events um here you will get that if i have another conversation as that be a key um and run the conversation you will see another event coming in and that event will be the vip what did i name it i was the vip usage trend event will show up and it will automatically create the trend line of that occurrences of the vip having a conversation and i think i created a event property which should be the conversation type so this will be the conversation type which happens to be the distribution of va uh only versus vega live agent escalation and some other properties i can create multiple properties that are related to the vip so this is how you would create your own custom events that only you care about that might not be included in one of my out-of-the-box widgets and so that's custom events another thing to know earlier question how do you change this self-soul rate definition that you will go to the formula override in formula override um you have an option to change the formula for duration self solve and state feedback so let's look at cell solve rates formula override this is a quick script value that you can change as you can see what we're doing is that out of the box and live agent transfer is uh not happening and the feedback is positive um we are calling that and the topic is actually complete state it goes complete we're calling those conversations to be successful if you want to change that definition you can do that coming here and change the formula and this absolute rate will change any questions that was great you good thanks uh we have one question about uh issue auto resolution from mary is the system smart enough to note the customer attempted to use to identify topic and was unsuccessful now i can probably answer the first half we do we can't identify the topic being used as to whether it was how whether success how would you uh how would you address that you go i wonder if are you asking if we are tracking a specific syste user id to see if that user had already tried a virtual agent yesterday and instead they decided to file a ticket the quick answer is no the premise here is that we assume a lot of end user just likes to file a ticket without even knowing the existence of the virtual agent um so whenever they go to a portal or some other myth as some other means we try to detect if the same topic exists in the virtual agent and basically this is the introduction to virtual agent for the end user who probably didn't even know existed and we will send the actionable notifications to slack or teams or you know that type of method you know messaging channel for them to complete their task yeah uh yeah it's a good point i'll just add that um customers would probably play a little bit of detective as well or you know analyst as well because uh uh ish issue auto resolution or iar do use virtual agent topics which we do have analytics on whether by default or any of them that you create so it is it is possible to determine whether uh topics were directed or opened via issue at a resolution and then from their kind of filter slice advice data to figure out whether they were completed and successful or whether someone got stuck um are there any uh if there any other questions feel free to type them out in the in the q a and uh yeah we'll see if anything comes in uh yeah we have the uh conversation analytics admin in the viewer role that we created okay so that's that's addressing katie's uh question do we have you need a specific role to have access to be able to edit the dashboard uh i i assume administrator yuko or anything else that you just mentioned yeah it's a conversational analytics um administrator rule is the only one you can add i did that great um great we'll pause here for further questions again the conversational analytics dashboard is available as a store app another thing we wanted to mention is that because the store app we also have more opportunities to get feedback from you our customers and our partners so if you have suggestions to improve analytics or any of our virtualization features feel free to again visit our community where we have an idea portal submit ideas and if you like other ideas people submitted upload them our product managers do read the idea portals uh mary asks is this available on-prem right yes right gaurav actually this is when i need it sorry uh can you repeat the question yeah yeah i can hear you yeah sure from mary is it is this available on prem uh good question yeah i guess the broadcast yeah okay can we check at the back yeah yeah i i feel it might not be because we will be uh transferring the data to app c and then retrieving it so it might not work on one of them but we can double check once yeah something we'll do is um when once a year once this video recording is available on youtube i'll try to add a comment or try to slip into the description but any question that we didn't get to address um another thing i'll mention uh is that these analytics are being refreshed at a regular cadence via a scheduled batch job so uh during your testing or during implementation uh you know data updates won't appear immediately or you know snap fashion you just wait a couple minutes uh for the job to run to again get all the data and adjust it and and present it to you on the dashboard just a heads up there uh great okay i think we're gonna wrap it up here unless there's any other questions uh let's see here yeah if there are no other questions again thank you for your time thank you for joining uh thank you for staying with us a little bit over again feel free to visit community.servicenow.com for all your questions and this session will also be recorded and available not only as a link on community service now but also uh on youtube all right great thank everyone thank you for joining and we'll see you in the next couple tuesdays thank you