all right i'm going to go ahead and get started with our announcements again thank you all for coming and attending the virtual agent academy as always i want to point you to our virtual agent and nlu community forum there you'll find the latest resources faq's troubleshooting guides as well as the ability for you to ask questions there and then give answers uh many of you long time virtual agent attendees are experts by now and you're you should definitely feel free to jump in there and provide answers and insights into folks there asking questions this session is recorded every youtube va academy is recorded and definitely feel free to take a look at the virtual academy playlist and if you like what you're seeing and you're watching this from youtube go ahead and click that like or subscribe button that way you'll be notified every time a recording is uploaded uh we i usually upload the recording within the week so definitely be a lookout for that coming soon we have more academies coming soon for you so you are many of you are probably aware we have a mobile academy a performance analytics or platform analytics academy will come in sooner or later an mlai academy led by va academy hall of famer nabil last week the first academy is on july 13th feel free to register at the also new ai and intelligence community at uh at the bitly link there ai intel com okay once again that's ai intel com or just go to the community forums and look up the ai and intelligence community and he there he has a link where you can register for the uh the first mlai academy on july 13th so today's topic we're going to talk about how to get user feedback from your virtual agent and agent check conversations so this will dive into the configuration uh for the virtual agent feedback as well as the agent chat post chat survey um i think i you know for for those of you i met at knowledge it's it's good to um see you again here uh while that knowledge i got a lot of questions and even outside knowledge saying hey how do i know how people are appreciating my virtual agent how do i get user feedback from va i'm going to show you how i'm going to show you how it gets measured in our analytics and and all that good stuff so here's our agenda and before i dive in one more announcement it's our 50th virtual agent academy and the emojis are freaky right now without the colors but i hope you understand what i'm trying to get at but yeah i was i was just taking a look at our playlist earlier and there are 49 videos on in the playlist that makes this 50 happiest 50 year virtual agent academy here's another 50 more so um to kick off the academy i wanted to show you the new virtual agent feedback ui that came in rome so so for some of you um who seen the feedback survey from virtual academy uh from virtual agent saying earlier releases you probably would have seen a you know older carousel with these large smiley faces that was like shades of green shades of orange shades of red well no more our feedback um component is now much more streamlined uh with emojis it's simply a good neutral bed right there and the feedback component also works in teams and slack integration so uh regardless of where your users are whether in web or on in the messaging app they'll be able to see this okay so uh we'll jump in now to the exercise uh this is a brief agenda of what the exercise gonna do so i'm gonna pause my screen share to uh switch over to my other screen with an instance so here i am uh just let me know if you don't see my instance but i'm gonna first uh show you all how to how to turn on the the feedback survey which is pretty easy um if you have any questions feel free to put them in the q a uh it's slot not the chat the q a uh i we will read both but q a's is obviously much more easier to just jump straight to your questions so i'm going to start as always in the ci home page so again the ci home page is where all our configurations located as of san diego and i'm going to go to set up greetings and uh custom greetings setup this is where i set up all my setup topics right so greetings uh survey uh search us fallback topics etc when i go to setup topics and one of the center topic type is survey and you just add the virtual agent feedback topic here so i would click into survey it's already configured here and i'm going to show you how you configure it click and click into survey and select the setup topic virtual agent feedback topic right here this is included for you out of box so you should have it already all right so once you do that you will have feedback at the end of all your virtual agent conversations so let's see what that looks like well actually before we do that uh let's dive a little bit deeper into the virtual agent feedback topics so to show you the inner workings i'm going to go to the home and i'm going to open up designer so the virtual agent feedback topic is a topic and we'll go a little bit into how it's built okay look up virtual agent feedback there it is it is read only if you wanted to modify for whatever reason you simply duplicate it but essentially only one um topic block the survey topic block and you see here these the survey id so um you know we'll put this in the back of your head we're going to revisit this later but you'll see why or how this is used in a moment so the feedback topic only has one topic block and then if you really wanted to you can go to this you can look up the survey topic block and there it is and uh it's much more complicated and a bit more high code so we're not going to go into it but it's there for if you wanted to take a look at that okay so yeah again more importantly keep that survey uh id in mind that is a assessment type so again we'll look at that in a moment all right so that's how you configure the feedback survey that's the inner workings of the feedback survey now let's go to let's let's demo does the survey now so i'm going to go to uh service portal right here i'm gonna start a conversation a virtual agent conversation of course i'm an admin that's why i have so many issues i'm just gonna run a test topic yes it is you can see this is my my knowledge demo environment and boom there's my there's my server how's your experience chatting good neutral bad uh i'll select i selected good plenty of times actually in the past unless like neutral just to keep the keep the data interesting so um that's that's that's every user who is va who ends their conversation naturally will will get the survey okay so now how do i as an admin or as the owner of the virtual agent see the results of the survey so let's take a look so i'm gonna go back to the uh the ci home page i'm not gonna go to the analytics dashboard so the analytics dashboard one of the outer box widgets or only available widgets is the feedback from your users so i'm going to the deflection configuration is not as important i'm going to extend the timeline just a little bit say to the may since that include data from all the knowledges you can see at the bottom here wait for this to load uh user feedback take a little longer to load than usual i guess this will take me a good time to take a look at questions uh will every user get the survey or it depends that's a good question we're going to address that right after this thing loads i'm going to pause share hit the refresh button oh there it is there we go user feedback you see i had you know seven good uh one neutral oops i'm gonna go ahead and keep moving forward i'll wait for that i'll leave that to load later there i've got other things to show you so that question that just happened hey is this gonna this certainly gonna occur everyone or just a random amount of people um that's configurable so i can set a probability of when or how often that survey um comes up and do that um it's a system property so go into here in the system properties but i'll show you the system property it's called and i'll i'll post it in a chat com.glide dot cs dot survey dot probability so i'm just going to look it up here and just look up the word probability and you'll see it and i'll post it in the chat as well give me just a moment there it is and you can see i set the value as 1.0 meaning 100 if i want i can set this to any uh number between or any decimal number between 0 and 1. i can set it to 0.5 that means the survey will only appear 50 of the time uh 0.33 point whatever you want so that's you set up the uh how often or the percentage that the survey appears okay cool so that is a virtual agent feedback so now the next part i wanted to show you all is the post chat survey after an agent chat conversation so let's say the user just spoke to the agent and uh you know you want to measure how good or how well the agent's doing you can have the ability to send the user a survey after that and let's let's show how that's working so i'm just going to soldier on here so the post chat survey uh very quickly it does require uh an agent chat plugin right uh not connect chat agent chat so let's jump to uh advanced work assignment under settings under cues okay so you may have more than one queue you know idtq h turks etc i only have one queue the agent check queue right i'm gonna click into that make sure that keyword is active service channel is chat and i'm gonna go to this tab here post chat survey okay so i click into here this is a record make sure this is active okay uh i have no condition so it's going to appear basically every time or regardless of of what have you and then here's where i select which survey i want so by default you will have a survey in here i believe this is its service service desk post chat survey this is actually an assessment so if i drilled into here you would see that there's an actual assessment metric type uh etc making sure that this survey is also active right and then uh that's basically the gist of how that's configured so i will show you how that works so i'm gonna go back to my service portal let me start a new conversation now i'm going to start an agent chat conversation so in my um uh in my other window i've got that angle as an agent you don't see it but you don't have to worry about that i'm going to just start a live asian conversation [Music] and i'll have that acceptable just say hi to each other and then i'll end the conversation just okay getting it over with and i'll have that type of pie i'll have the agent type hi everything's hunky dory and i will have beth again end of the chat and i get a survey how satisfied uh very dissatisfied i'll say i'm very satisfied and this is basically a survey that's been transformed or converted into um the virtualization conversation flow as a one question survey which is all good but let's say i wanted to do a bigger survey right or a survey with more questions i can do that so let me just start another conversation uh but i'm not going to start the conversation quite yet i'm going to show you how to change the post-test survey into another survey so you know again a couple things to keep in mind so first got to make sure this is active you got to make sure your survey is configured here and that this survey is also active i'm going to click into here so this survey is technically called an assessment metric type i know it's a mouthful but that's what that's what the surveys are called in servicenow make sure that survey is published it's active and the other way to make sure that that field let me go back for a moment that this field shows you surveys oh let me uh you don't see this pop-up but if i were to click into magnifying glass i see a list of assessment metric types that i could uh select from you know what i'm going to stop sharing and resharing my desktop so you can see that pop-up give me just one second all right see the pop-up these are all the list of surveys i could use as a post chat i could even technically use the the feedback survey that i used in virtual agent um don't have to but i can um the question is like how do these show up right first off before i go even further than that again these are assessment metric types this feedback survey remember that this idea showed you in the survey topic block that is how this survey gets configured into that virtual agent feedback survey so if i wanted to change the survey uh for my virtual agent survey i could have selected a service desk satisfaction survey or the priest this pre-chat survey this put you know i could have selected any one of these uh with it to configure into the topic block uh but that's what's configured out of box and read only that's how you can change the va survey um i i don't recom i mean i i don't recommend it but i'm not going to stop you you can totally do that if you say you know what i want more insights in my virtual agent surveys i could do that you you can do that and the reason why i don't recommend it is because our autobots dashboard our analytics uh uses this feedback survey that the smiley face survey um i have yet to experiment how that analytics would measure or or work if you were to change the server that's why i don't recommend it that said obviously i'm not gonna you know i i wouldn't stop you so how does that list get popular let me go back to did i okay let me go back to this and go to the assessment metric type record active publish and this check box has to be checked chat survey if this is not checked then this metric this survey will not appear in that pick list i just showed you earlier okay so that's what makes uh this survey configurable as a post chat survey okay you can see the metric categories here um this here's the question right how satisfied were you as a choice and if i were to click into here you would see actually the configuration there i'm not going to do that if i wanted to see i can see okay this was run a couple of times last week once today right and then um if i wanted to see the results i just click view score card and there's only one question yeah satisfied very satisfied this this was the results right so that's how you how that's how you number one um configure which survey to appear the process survey and number two how you ensure that that survey appears in the pick list is available for selection and actually shows up so again published active chat surveys checked all right now last thing i want to show you is how oops i'm going to go back to the cues last i'm going to show you i'm just going to change the post chat survey to another chat survey just to show you just like another possibility right and how to do that go to the post that survey tab go into here again make sure this flag is active as well and i can change this to um uh as a service desk satisfaction survey okay i'll click update so now i have a different survey so i go to service portal i'm gonna connect to a live agent again [Music] give our salutations i'll have beth and the chat [Music] and now you'll see it's a different survey depending on how you configure this survey it's gonna render differently right so some experimentation is is um required just to make sure you have a good experience this looks all right the font's slightly different but not a huge deal you can just um start answering the survey again this is these are all choice um question types or assessment metric types to use service pronouns terminology and you can see this is just a longer survey i could skip questions if i want um you can see in the next couple questions there is an ability to yeah i can type you know all those good i could you know there are open ended questions as well click send and that's the end of that click um chat survey if i wanted to once again i could i'm just give this a skip and just jump right to that survey from here you can see the response trend view the scorecard to you the results here and uh yeah many more questions and you can see the results there keep in mind if you use this survey in more than one location like both your virtual agent and for somewhere else you'll see the results combined so uh you know just for as a good practice you just may want to make sure that you keep your virtual agent and your agent chat and your other types of surveys uh separate so that you can have you know you can more properly measure the results there cool so that's all i had to show i actually have one more thing to show so this is uh optional slash advanced some folks have asked well you know enter uh how do i link the uh particular survey result to a particular interaction and there's no easy way to do it i'll show you the technical way to do it okay and the way to do that you can see here these are my more most recent interactions here with beth if you go to the interaction record uh you see the transcript here there are interaction related records so if you were to configure this page uh one of the uh options is called interaction related records and here you can actually view your assessment instance so every time i take an assessment i took a survey an assessment instance is created right like this and if i click into here this is essentially the uh basically the survey and the survey results the value uh the string value if you if you typed in a question and so that's the link how you hop from an interaction to the assessment instance um yeah here's a trigger id interaction id basically here's the assessment instance id essentially so that's how you can link that so you can see the relationship um it's again it's not an elegant um uh relationship and you know i understand that we're looking into that but you know if you wanted to to do technically and drill into the details that's how that's done awesome okay now i'm really done so with that i'm going to turn it over to questions uh just check the chat menu real quick yeah victor we've had a couple in the q a um i don't know if you want me to read them and we go through a couple but there's there's been some great questions so far and configurability of surveys um yeah let me let's have a look a fair uh fair warning i am not a survey expert or i'm not an assessment expert so i'll i'll do my best okay great so so this one's really about um the system property where you set the percentage um to the point where the surveys get a trigger and the question is does it trigger by user meaning if you have a system property set to 15 50 is it gonna trigger for that individual unique user every other time or is this for all concurrent users and again if you don't know the answer we can take this as a follow-up and post a community i good question from experience it's it's per like interaction meaning you know if one person hit the virtual agent like 10 times on average they'll get the survey five times it's not like oh fifty percent of your users will get in that's just fifty percent of the times someone hits the virtual agent okay great and the other question was around configurability of surveys and do you have any recommended practices so if you want to have different surveys going to different populations maybe it's surveys going to different role based groups or different geo locations do you have any recommendations for best practices there um i i you know there's there's nothing too crazy to consider of course as i mentioned you want to make sure that your surveys don't overlap because when you annualize the results you want to make sure you're analyzing eating chat surveys you want to make sure you're analyzing virtual agent surveys separately um the other thing i mean some things consider obviously as i show you each cue here has a post chat survey i think that's a pretty good way to to do it like you want to make sure that you measure the the performance of your it q or your i.t group and versus say an hr or legal queue separately just to make sure that you know your the each of these queues or each of the 8 and chat agent groups assigned to these keys are doing well other than that nothing um again too crazy the last thing i will say from experience is that um there are many many question types you can use i believe it's like 12 plus um some of them may not render well in virtual agent um for example i know you can do image scales i know you can do a variety of other question types so definitely experiment and test and make sure that the survey renders well in virtual agent choice works very well as you saw open text works very well as you saw so definitely if you're going to do something else or do another question type you know kick the tires give it a test yeah and victor just to add to that and that's a great point is um really be mindful of designing for your end users so if you want someone to easily and quickly complete a survey you always want to make it really visible really easy to do and then the other thing to remember too is when people complete a survey they expect a follow-up or an action so what i've seen some organizations do that works really well is have immediate follow-up if there's a negative result or response from a user and just have the internal team follow up with that user and collect more information around that so it usually makes people feel really good and helps turn not so great experience into a positive one yep good point [Music] and then one other question does virtual agent feedback score data include the virtual agent topic itself um unfortunately not at least not today um if i were to click into yeah there's no well actually this is what you can do i can actually i didn't show you this because it wasn't loaded for me but i can view let me do that again because i might have been too fast um you can click into drill into the the good the neutral the good in spanish the good um responses if i click into it i can see the conversations that actually were good that i can view the transcript i can view the timeline so that's how yeah you so the short answer is yes a bit of manual analysis but the short answer is yes i can link or relate good conversations to the topic to transcript uh the conversation details et cetera in the analytics dashboard great i think that's all the questions for today awesome well thank you all for attending i will see you all again uh in a couple weeks for virtual agent academy again the recording will be uploaded for uh in youtube hopefully by the end of this week thank you all for joining i'll see you next time