all right the it is one minute past the half hour mark good morning good afternoon everyone thank you for joining the virtual agent academy let's get started so uh before we get started please take time to visit the virtual agent and nlu community there you can explore best practices read up to up-to-date content and ask our experts questions as well as the community questions to find our answers about your virtual agent and and nlu questions rely on the experience of others and get your answers if you're watching on youtube and you like what you're seeing feel free to subscribe to our uh servicenow community channel and like this video or leave comments below so today's topic is expanding your nlu vocabulary for effective virtual agent conversations i'm victor chen and with us today is also david van gosda who is outbound product manager for platform and now intelligence so our goals for today we are going to go over advanced nlu and then dave is going to go over an exercise on nlu vocabulary sources and we'll wrap up with q a if you have questions during the meeting or during the demo feel free to use the q a button on your zoom link before we get started i also wanted to give a shout out this is a virtual agent academy we also have a mobile academy on alternating tuesday mornings or afternoons if you're if you're elsewhere in the world and then every third wednesday of the month every third or fourth wednesday of the month we have an hr academy so again please visit the mobile and hr communities for more information register and they'll they also have events and webinars just like this so let's do an overview of nlu and advanced nlu first so in quebec we released an nlu workbench which you may be familiar with that reside in the studio while it's now residing in a platform itself so no longer it's no longer in studio and it is a no code interface that allows you to do an nlu model authoring management and tuning you can train test and publish uh nlu models directly in service now in a no code in a no-code environment we in quebec we've uh supported additional languages we support up to 13 languages now previously we may have supported fewer but now it's it's many more languages you can search for your nlu models as the amount and breadth of your nou models expand we have pre-built understanding of software and hardware terminology we understand that software and hardware especially for iot sm customers is really important for nlu and we've actually pre-built that for you so you don't have to define those yourselves you can build vocabulary from tables or lists so rather than relying on a lot of manual entry for your utterances and entities you can rely on vocabulary from your existing data and that's going to be the main thrust of our exercise today and finally you can also have your entity values mapped directly to tables as well so again less manual entry when defining entities such as your list entities advanced features so we do have a store release available as a well certainly as of april uh for nlu advanced features and there you can optimize nlu even further for for greater of virtual agent experiences for your users so you have an intent discovery so that uses uh itsm taxonomy which is uh something that servicenow created based on the past data and experiences of our customers and we actually cluster your incident data or your or any other incident related data to recommend what intense and virtual aging top topics your organization should uh should implement so if your organization has a lot of uh incidents on whether it's vpn printer issues email issues etc we can recommend what virtual agent topics for you to activate and and to use based on that data we have conflict review so we understand that again as your nlu models and intents expand for your organization we want to give you tools to find potential conflicts between intents before you go into production so if you have an enter if you have an utterance that is surfacing more than one intent we don't want your user to experience that and our conflict review feature surfaces that for you for you to resolve we also have batch testing that'll give you insights if you're in a uab model still predicting that peak accuracy so you're able to upload multiple utterances and and testimony once with the nlu models you have to see what models and intents are succeeding and which are perhaps falling behind and that again allows you to improve and engage and see what intensity need tweaking or what which models need tweaking and finally we do have nlu performance recommendations and that analyzes your ongoing virtual aging conversations offers improvement opportunities and again make sure that as your organization uh moves forward and as it expands your virtual aging capabilities and niu capabilities your models and intents uh hold up and there are no changes that you or there may be changes that your organization needs to make sure that you're serving your your users and customers the best way so again all this is to serve that feedback loop that we want to provide you as guidance you want to analyze and design build operate and optimize and then just iterate and make sure that your that your nlu models are constantly catching up with uh the growth uh and the changes of your organization in in and we provide those direct easy loco no code tools for you for you to do so note that the advanced features i mentioned on this slide are do require the your pro or enterprise license so itsm hr pro so on and so forth without further ado uh we're gonna i'm gonna switch over to david he's gonna do an exercise on vocabulary sources uh david feel free to take it away thanks victor i'll be walking you through an exercise where the objective is to create more effective virtual agent conversations by expanding the vocabulary for miu using vocabulary resources and mapped entities it is possible to more easily understand what the user's intent is you can use this information to skip steps in the conversation providing a smoother flow for your end users i'll start by showing the current experience of requesting installation of a new software application so we have our service portal with our virtual agent and we have a series of topics that we cover and one of them is the software related request if i request that topic then i'm going to end up in a series of questions and answers where where what kind of software i would like to do and what what do i want to do with that software in this case it's install software update software or remove software i choose to install software and then in the topic we point back to an internal servicenow table where we show all the software that we have in the catalog and that is a list of applications and in this case it's it's you can see that the list grows and grows so at some point it will be more interesting unless we want to use the search feature for example to have a better way of finding the application but i can choose zoom and continue the flow so this is a pretty decent workflow from here but what if we want to make it a little bit smoother well the first thing that we can do obviously is skip the step of skipping so i could say something like need to install zoom and then if the topic is correctly found then the next follow-up question is can you install software update software so i still need to start answering the questions from there but at least i don't have to pick from a list of topics which could also quickly grow in your enterprise as i said this is uh we're looking at software applications here but you could think of many more items in your environment uh that could possibly be candidates that you would like to basically capture uh using the the the the in the within the utterance all right so where do we start improving this experience for the end user by for example better understanding the utterance in the first place let me go back as i end this conversation let me show you the list of software applications that is in this environment so here you can see those 17 items that you just saw in that drop down in the in the virtual agent as well or in that list you can see that we use let's say commonly known applications blink reich box microsoft teams webex etc so that is just a table uh standard table software catalogs uh all active inside the instance now how do we improve use nlu to improve the experience on the on the end user's side in within the flow well as victor mentioned right we we have now have the new workbench which sits in the in the menu now has a nice little menu entry and here you can find all the related features and utilities to that workbench so we have the models where you can see all the models with all the languages and you can use them to basically sort and search through your models we then have the vocabulary sources and we'll dive into that in more detail and then we have some properties related to the niu workbench or the applications and then we have those four applications that victor also mentioned the niu model performance conflict review batch testing and intent discovery so they're all part of the workbench so if you type in nlu workbench and you have activated or installed from the store the end both the nu workbench and the nu workbench advanced features then you will get to see all these entries uh i'm first before we dive into the vocabulary sources i want to show you the nlu model that we're using here with that software request which has a mapped object to the virtual agent topic this is a nice improvement also in quebec where you can now move between the virtual agent topic and uh the corresponding intent on the niu side both ways so both from within the virtual agent topic as from the nlu intent so that makes let's say navigating and administering the whole setup a little easier if i go into this intent i have a series of utterances that we use to basically train this intent and as you can see i have things like dropbox zoom webex microsoft teams reich applications that you saw in that list that i showed earlier so if i test here i need to install zoom i'm going to get the prediction and the reason why that topic was also picked earlier in when i had the virtual agent in the service portal so we can see that the software request has a 97 accuracy so yeah that's the reason why it got picked nothing we're there um what's important here is that yes this is the way to do it right we previously had to basically add utterances um or update the the vocabulary with a lot of the the software that we had in our environment but that was all let's say a manual labor so you would have to create different versions of your utterances different shapes and then add the software applications there in order to use them as for example list entities that we had prior to quebec with quebec that is slightly changing but before we go there we first want to make sure that we let's say use the vocabulary and the capability now with the vocabulary resources to quickly improve the the the knowledge of uh the vocabulary from existing information in the instance okay so let's do that so i type in nlu workbench and i go to my vocabulary sources and i can see that we already have some that are not synced but i think at first we want to just show you how it works so let me just point to a table where we create this vocabulary resource to in this case i want to map it to that software catalog so that's the pc software cad item and then you have to put in a handle the handle you use in inside your utterances to refer to this vocabulary source so i'm going to name that software app and then you have to provide a short label and the short label is also sort of the synonym so be mindful of that as you uh create that because that's how it will be let's sort of replace when when training the model so whenever it comes across a handle like software app you could also think about it as that replacing that with the the short label so software app and as you would see that in in the version that or in the vocabulary source that we will be using in the exercise we will actually be using the full name software application so be mindful short label give it a sensible name that you think that really captures the group that you're that you're referring to and that will not interfere with let's say existing synonyms in your vocabulary so we choose something that's um unique so as you can imagine and and if you remember that i in the in the in this software catalog table all of the names in the display name were called uh things like zoom application webex application that is unique those were all unique and they're very descriptive and you would want to do so you want to make sure that your source tables and the the columns that you're choosing the uh you're building the vocabulary source on are also built that way so you might have to do a little bit of data cleansing to make sure that the data that you're pulling from in is unique and descriptive at the same time in the advanced options you choose languages and um yes with nlu you're not only you're creating let's say a model and vocabulary resources and all of the artifacts the vocabulary per language so if you have multiple languages in your environment you're going to create let's say artifacts for each and every language so if you have another language like french for example then you would create also a french nlu model you would create a french vocabulary source and you would create a french vocabulary mapped entity etc etc that is a bit of work obviously but it's currently the way uh to do it with uh the the languages we're looking at ways of making that easier and less administrative in the future i just want to make sure that everything is active so i'm going to set active to true as i enter the query so that will limit the the items you can just use the normal query builder to reduce the number of let's say items that this vocabulary source refers to and then we have the refresh and the refresh is necessary because this nlu vocabulary source cannot remain just inside the instance the data that it pulls into the nou model needs to be part of the nlu model and the reason for that is that at prediction time you don't want the model to have to reach out back into the instance and do a lookup in a database for us a specific entity or a specific vocabulary mapping so that would significantly increase the prediction times and that's the reason why we're basically synchronizing the data and that's also why you have to keep in mind is this vocabulary resource a table or a field that frequently changes if so you might need to change it to some arbitrary time as you can see we're not saying you're going to run this every minute right that because that would mean that you're trying to create something on a vocabulary source that's constantly changing changing and that would mean that you have to retrain your nlu models etc so that's not that's not going to be a very good practice so consider this as you build vocabulary sources do it on relatively infrequently changing tables and columns and say then look when you can do that every seven days and obviously you could say every seven days and then you could run it manually you could basically trigger it also manually but yeah just keep keep that in mind fuzzy matching fuzzy matching by default is on which allows slight misspellings and partial matches which is great but it also comes at something that you you need to be aware of right it could mean that for example i need to install zoom or i need zoom would also match uh zoom with i need room because room and zoom are yeah not that far apart in terms of characters so keep that in mind there are ways to basically um deal with that for example by uh just making sure that the utterances that are part of your intent are uh specific for that intent and that the intents that are part of your model uh are not that generic and broad in in the sense that you basically try to cover uh both requesting rooms and requesting zooms in the same model because vocabulary sources are restricted to a specific intent or a specific model but never across multiple models okay having explained all that let's just save and because this takes a little bit of time i'm going to skip a step and i'm just going to go to the already created vocabulary source which is the software application and that one basically has the synonym or the short label called software application so basically wherever we refer to at software application you could basically replace that with software application it has the exact same settings otherwise okay now let's look how that how we use that inside the model so i mentioned that right we before we had all these software names inside the utterances so that's exactly what we need to start changing so instead of saying install dropbox we're going to say install and then we hit at and then with app we get a selection of vocabulary sources in this case we have only one so i'm going to choose software application and i'm going to hit save and then i'm going to search through my rest of my utterances to see where else i should be making these changes we'll update those accordingly all right and then we have two more that need updating there we go almost done and before we can test this we obviously need to train so we're going to train this model so that the model is up to date and it has pulled in all that source information from that source table that programmable resource table combined it into the model and then combined with the utterances in order to make successful predictions now the easiest way to test now is because we already have a published model we could say something like i need to install zoom and we might get two predictions one with the train model predictions and one with the published model predictions and then we could say something like hey i need to install i need to install dropbox you could pick any any of the the ones that were in there i need to install teamwork and you can see that although the the predictions slightly differ um either above or or below they're all well above the threshold and the reason why even without knowing the the let's say the the vocabulary source and still predictions are in the high 90s that is because we're doing software requests here and one addition to in in quebec that we introduced was basically a a software vocabulary and hardware vocabulary addition to to the vocabulary automatically so we already know a list of over 200 software applications that we use to train the model for so that's why the percentages here are already high if you would use something completely different like your enter your enterprises meeting rooms for example then you would see very different results there okay let's publish so we're good to go there um so we now have let's say successfully trained and published the model with any of the software applications that we might have in our environment so whenever someone has an utterance which refers to something like i'd like software application to be installed right remember that synonym that's going to be a significant successful prediction and it can be any of those software applications right although also the ones that are are less well known and and i think that is that is the uh let's say the big differentiator before quebec and after quebec with the use of those vocabulary resources however it's not the only improvement right as i mentioned if we start now to introduce mapped entities and mapped entities is actually the replacement of the list entities then we're going to be able to let's say provide that better user experience when in the topic flow because we're able to predict which software applications the user is referring to so that we can skip steps in that in that topic flow so let's show how to do that so you basically select the the vocabulary source entry and then you click mapped entity and you say create new entity as you can see we now have this basically grayed out and it now maps to the source which is a table or a list and not the the let's say the the self-created list of values which we previously had with what we called the list entity prior to quebec we can make it available to the entire model if we want to so if we have more intents that would use this potentially this this software application then we could check this box and we use the same map value for the entity so now that we've done that we need to basically click all of the utterances where the where this reference is needs to be mapped so let me just go ahead and do that and once we're done we can train again to make sure that the training and i see that i forgot one so let me wait until the training finishes and then do that one as well to make sure that we've covered each and every one retrain again and then we'll test the utterance again so i need to install zoom now as you can see we now have a software request but we also have that entity software application right and it refers to zoom application and what you can see here right it doesn't it didn't pick the uh the string coming from the utterance oh what it did is basically it took that display name uh from uh the the source table and uh place that there and this way you're actually making sure that you have uh the same outputs going into the virtual agent when a prediction is returned so uh regardless if people make a mistake or not you can basically ensure that you have a consistent value returning to the virtual agent so let's see what what happens in our virtual agent environment so if i go back to the service portal and i start a new conversation i want to install zoom you can see that nothing has changed and the reason is we did not update the topic yet so we need to update the topic so let's make sure that we do so as i mentioned we now have an easier way of getting to the topic so actually i could just click this and it would open up that topic in the virtual agent designer and i can go to the flow and i can see that in the flow we have that software request then a decision and whether or not we can install or update and if we want they want to install certain software we look up at a table and that table is that software catalog and that is the one that we want to use because we know that we're we've just created that that software application i just need to make sure that i publish my model that's a good one so what i didn't do i did do training but i didn't publish so the virtual agent never basically doesn't know about unpublished models so i only trained the model i did not publish it so i first have to publish it then when i go in to the virtual agent and i request refresh my list of intents go into the flow go into install software in my list of entities i now see my software application right so uh yes that is something to be aware of it the topics only look at actual published models and that's very helpful because sometimes you're tuning your nlu model and as long as you don't publish your model you can still test it but as long as you don't publish it um nothing will change in in your production environment so that is a good to know i'm not going to enable and know you and i don't want people to basically be able to change the conversation at this point i don't need to i just want to skip the confirmation if we're able to identify the software application so let's publish that and a test again so i'm gonna go and say okay install software and so if we type that same utterance here so i need to install zoom what we can see is that we first have to enter still if we want to install update or remove the software but if we choose install then the application is automatically selected because uh the in the utterance zoom is picked up as an entity um that result or that that prediction is being returned to the virtual agent as zoom application within the topic zoom application is being used to look up uh the respective record inside the source table which is found and automatically selected and that is a great way of basically reducing the number of let's say clicks by your end user or selections by your end user and there's one more thing that we can do to improve right because we still need to basically pick install update or remove so can we get rid of that step yes we can so let me show you that in the final step of this exercise all right we're closing this off and we're going to go back into the intent within the utterances we see things like install so could we use these to basically categorize a utterance in a specific category and yes we can and we use the same mapped entities as i just talked about earlier but now we use the mapped entity where we have a self-defined choice list and that basically is the same as we had already prior to quebec which was the list entity so i had pre-created the software action mapped entity based on a list and the list choices that i made were install software remove software and update software so in this case we choose install software and we're going to go through each and every utterance where we see install that basically is an installation type intent and we're going to select that utterance sorry that mapped entity then we're going to train the nlu model again and publish so that it becomes available for the virtual agent and we can then use it inside the topic so we trained it we're publishing it let's make sure that we test it i need to install zoom and if i do that i can see that we now match to two entities not only the software application but also the software action in this case install software so if we now use software action also in a step in the topic flow then we have the chance to capture more of the user's intent and automatically skip steps okay in the topic flow we have we start with the software request where we basically look at install software update software or remove software so this is where we're going to use that newly mapped entity so we select software action from the associated entity remove remove this because we don't need to enable nlu but we do want to skip confirmation for that recognized entity so that whenever install software is returned with a prediction it automatically skips that step and goes right into that decision right that decision is made here on the install where you can actually see that when a software request is install software then go down below here into that picker where we have that software application entity all right one step left to publish this and then basically start a new conversation type i need to install zoom and see that immediately the zoom application was selected from the backend table and no need to enter install update or remove this concludes this exercise so now we'll continue to the remainder of the presentation what remains is some additional information so where can you look up uh relevant uh let's say informational sources uh in the documentation obviously uh and on our virtual agent and niu community it's good to know that knowledge 21 is right around the corner and there's going to be a course there which is called getting started with the nlu workbench which you can then go through each and every exercise amongst one is the one with vocabulary resources but it also touches upon all of the advanced features such as intent discovery and the other advanced features and then there are some related nlu academy sessions getting started with entity recognition and niu in quebec you can find those in the playlist for the virtual agent academy on youtube having said that i want to thank you for your presence and i wish you a very good day and see you next time you