so as always feel free to check out our virtual agent nlu Community uh go there to find all the latest content we have and go there to get answers from our experts and uh all of you by now are probably expert enough to give answers yourself so you know there's always there are folks there asking questions so definitely check out our community site it's got a bit of a a Rebrand or not Rebrand but a a redesign of the UI hopefully things are a little bit easier to find uh but uh yeah check that out if you're watching this view a recording so all our academies are recorded um if you like what you're seeing go ahead and click that like subscribe button that time every time we upload a uh virtual agent uh or sorry CI Academy video you will know all right new name uh same awesome Academy some of you may know this as the virtual agent Academy it's now we're calling it The conversational interfaces Academy so the reason why because we're covering a lot more things than just virtual agent nowadays we're covering uh agent chat or coupling we're covering sidebar as well as well as virtual agent nlu like today so uh we just renamed it there no further action for you needed uh you know the fact that you all join is is good and you know I thank you for that again so again tell your colleagues spread the word foreign we're going to talk about uh something that I know folks have been asking for every once every once in a while we just need that you know it's like a muscle right we need to keep training and we're going to talk about how to tune nlu for the virtual agent that that way your nlu and your virtual agent can act uh more intelligently could get your users uh what they want the way they ask it and with me today is our nlu expert nilima Desai you've seen her before in our other academies so nilimo welcome and uh yeah let's take feel free to take it away okay that uh thank you Victor I'll go ahead and share my screen um okay um everyone can see my screen yeah thank you for joining everyone like Victor mentioned today we will go over NNU tuning tips for virtual agents share some best practices and um yeah you know on tooling your nlu models uh so our goals for today we will spend about 15 minutes to do an overview uh with uh or in the slides and then we'll follow it with exercises uh the exercises uh basically will you know kind of reinforce what we uh go over due to English overview and Then followed by q a link so uh coming to nlu model planning our best practices for virtual agent uh you know there are a few uh important uh best practices we've outlined here I'll just go over them one by one uh this one is very obvious starting with out of box models whenever they are available as we know every uh few months you know we have store releases with uh new uh virtual agent conversations and a new uh model intents you know that are available so uh we always encourage our customers to start with those and then in addition to that also to be aware of each release what the new and the new related features are for each release you know and then kind of being up to speed on those and um you know taking advantage of them uh next is you know using topics and intents uh for actionable Content you know so uh what we mean by that and then using an search capabilities for the rest of the content what we mean by that is like within your virtual agent topic if you need uh to handle actionable things you know like an update to a table right or lots of inputs from a user to get to the resolution of the you know conversation right or trigger a workflow then yeah by all means you know use um you know topic and intents but for the rest if it is just if all you're doing is serving a search uh you know a knowledge article then uh use search for those you know instead of building topics and intents uh this one is also important you know it kind of complements the previous uh bullet it's uh basically you know always constantly looking at your uh knowledge uh that you have uh you know it's basically uh and and serving it you know with uh creating that content right and filling those knowledge gaps on the instance um uh this one is again you know it's an important one that uh is easy to overlook you know uh skipping uh creating intents that can be self-solved with AI you know and what we mean by that is uh you know using AI capabilities uh in terms of uh as we know AI search is out now right uh and AI search has some Advanced capabilities in terms of uh you know genius results that gives actionable uh search content uh q a you know which gives targeted answers for a user's question uh so leveraging all those you know and in addition AI search has also you know a lot of tuning capabilities where you can uh tune the AI search model to you know kind of give certain results boost certain results suppress certain results things like that and the other one is nlu entities right so choosing nlu entities over creating separate intents and topics you know uh so what that does is basically it helps with entrant consolidation and you know overlap in your intents and we'll go over those scenarios during the lab uh during the exercises and also nlu entities helps with a better user experience right because if there is an entity match then you know you can skip those questions within your virtual agent flow Now using vocabulary items and sources in your model to teach it you know to make it understand uh unknown words you know and so we will go over vocabulary uh also in more detail uh later right and then testing and tuning your model at every stage this is an important very important step you know just to make sure that your model is not drifting and it's performing as for uh how you expect it to perform and then finally adhering to the recommended guide rails that we've outlined over here and there may be even more so you know kind of keeping those in mind so uh next we wanted to highlight the nlu tuning cycle you know this is important and overall uh process of building your models uh obviously first as you know we start with the planning stage here you know we have um uh great tools uh in the platform like intent Discovery or topic recommendation that can you know help you with your uh tuning uh planning process you know in terms of understanding what are the top uh intents that you need to focus on so definitely leverage those uh then comes the building phase here again we cannot emphasize enough you know where uh leveraging out of box content uh as much as possible wherever available and then you know once your model you have a good model uh testing and tuning iteratively uh is very important you know uh it's basically you want to teach your model to perform the way you need it to right and then once the uh quality is acceptable running via testing and tuning you know you can plan to deploy the model in terms of acceptable quality out of the box you know we recommend at least 80 percent correct uh predictions in the model so you know you should we do recommend that you know you get the model performing at least at that level or better before you deploy it and then once the model is deployed you know in the monitoring stage uh there are several uh capabilities available like uh the conversational analytics dashboard and a new performance dashboard you know which gives you a snapshot of you know correct predictions incorrect predictions uh and also uh very important is the unsupported utterances right so these are the utterances that did not trigger any intents so those you can look at to understand uh are there any new intent opportunities for your model or are there any tuning opportunities right if an intend didn't get matched then and it should have then you can tune the model so that's where expanding and improving your models um comes into picture so that's that's uh the nlu tuning cycle uh next we will go over some specific guide rails uh while you're building your model to be aware of uh one is handling unknown words in your model right uh making sure you know that you don't have typos in your in your train model utterances uh you don't have grammatical uh errors in your trained uh model samples right and here we can see there are some examples of that um in the right configuring vocabulary uh you know to understand unknown words or acronyms or industry specific terms right so that will make your because the uh nlu goes by the language what is is in the dictionary right so if there are words terms that need to be handled uh vocabulary can be set up for those like examples itdl you know information technology distribution list Etc um restricting your uh train samples you know we have seen occasionally sometimes customers will put in an English language model they might put a uh you know Spanish version of the train sample or French version you know so that is something to be mindful of the nlu looks at the language that the model is uh you know for and then understands only in that language okay next is handling model ambiguity uh here you know uh overlapping intense you know I touched upon it in the previous slides you know where uh you can have uh instead of having like you know intents that are overlapping with each other uh consolidating those intents and there is uh conflict review uh tool that is available you know in workbench that you can use also to identify those and we will go over some examples also in the exercises uh having uh not having low context utterances uh in your train samples in the model uh you know these would be like uh samples that don't adequately describe what are you what they you know issue is right so like if you have passcode or you know session or you know single words or even not not enough context in the sample right not having combined our process is in the train sample you know that's a No-No you know so if you are if you have a train sample that is requesting two different things you know within that same sample then you know that is not a good idea like in this example I need a new laptop and a printer right so we need to separate that into two separate uh utterances the next one is intense structure uh you know following the recommended intent structure um here uh you know in terms of quantity uh following the minimum requirements so for intense the minimum is five intents per model is the bare minimum that you need and up to 300 is supported but we don't recommend that uh you know so if you do have 100 or plus intense uh you know you can take a look at you know how you can consolidate your intents in the model right in terms of uh quantity there is also what is important is you know proper and proportionate intense size within each of the intents you know so uh for each of the intents you know minimum we need is 15 samples uh and the system supports up to 200 but again that is not recommended and if you do have more than you know 50 or 60 or samples in the intent you know you can take a look at uh you know kind of scaling it back see what is uh redundant and you know handling that and then the quality of the samples uh within each of the impact quality of the utterances which within each of the intent you know making sure that they are diverse you know in terms of how end users might request the issue and uh in terms of handling the different terminology and Concepts right that need to be covered at the same time you know you need we need to be mindful of avoiding duplicates you know so uh you know so that the sample size the utterances within the intent are not no that that big right so now uh next we'll go over using vocabulary in our models uh so here we've highlighted like vocabulary Concepts and then we'll go over different vocabulary types in terms of Concepts we thought it was important to highlight you know when to use vocabulary and when not to uh right uh vocabulary you want to use it when uh there are words in the English dictionary that uh you know that are not in the English dictionary that need to be a handle like if we have an acronym right or if we have unknown words right of we have domain specific context for known words right so that's when uh setting up vocabulary is very helpful to the nlu you know for the model to understand uh when when do we not need vocabulary right uh if you just want to handle case sensitivity right uh there is no need for vocabulary for that you know that nlu is case insensitive so you know you don't need vocabulary uh similarly you know if you have like a group of words that you wanted to um mean the same you know but these are all known words right so you don't need vocabulary uh you know don't need to set up vocabulary for that you know like if you have like problem issue defect you know these will be understood by another you don't need vocabulary okay uh and then understanding what how synonyms work right uh that is very important part of vocabulary synonyms is basically it tells the NMU nlu model uh you know how it needs to interpret the vocabulary that's set up uh and synonym will always be a known word in the dictionary right and it applies to all three types of vocabulary uh you know which will go over uh and uh there should only be one synonym so it's basically synonymous how do you want nlu to understand what that vocabulary is right so it's a known words single term uh and an uh you know a dictionary uh value right so next we will go over what the different vocabulary types so the very first one is vocabulary item so this is what we see within the in a new model uh the vocabulary tab right uh so that is the reason it's in the within each model is vocabulary item applies to only the current model right whatever you set up in uh as a vocabulary item and there are two types of vocabulary items one is regular and one is pattern uh you use pattern when you have like multiple variations of um uh the term the unknown term you know that need to be how number and we use regex for that and then for regular is you know where if there is only one term that is unknown you know that needs to be handled then you can use regularly right uh in terms of uh vocabulary sources uh there are two types one is list and one is uh table vocabulary sources list vocabulary sources are used to handle like multiple values for a single uh synonym right and within those multiple values if there are even variations you know within the multiple values and they those need to be normalized so that's when you use uh list vocabulary sources right um table vocabulary sources uh quite obvious you know it's basically you want to associate the vocabulary source to a servicenow table on the instance uh and uh you know so you can use uh table uh vocabulary sources for that when setting up table vocabulary sources it does need to be synced manually for the first time and then later on the system will refresh the sync you know periodically um and it's you know basically again same thing you know you use it to normalize the value so from the tables right and for all of these you know we can't hop enough that you know you only need a single synonym uh for the vocabulary but be it vocabulary item so so uh I'm sorry list or uh table right with table vocabulary one very important thing to remember is you shouldn't have two different vocabulary sources set up for the same table and column uh profiles with two different synonym values because that will end up in confusing the model okay so next we will go over exercises um and the exercises are basically designed to you know kind of uh uh just basically reinforce what we have um covered in these slides uh so I'll just go over one by one uh here I have this model um which we've created like purposely a bad model uh with some of these examples so we will just go over one by one uh the examples to see you know what those uh bad samples are so first we will handle unknown words um in my model I have um Outlook General issues and within Outlook General issues I have uh you know the very first one right we have I am getting an email error so I am is not a no word in the English dictionary so this should be either I am or I a password right uh next we'll go over reset password um in in reset password we have uh so many of them are password uh fully spelled out but then we also have PWD uh in many places uh now uh this PWD is not a word in the English dictionary so we need to tell nlu how it needs to interpret the word PWD right uh similarly in Outlook phone setup we have um you know some unknown words that are being used um yeah an example is um [Music] yeah how to use Outlook app on phone right now app is not a known word it's uh basically application uh so we should not be using you know these kind of short forms in the train samples so next we will go over uh creating a vocabulary so that these can be handled uh so in vocabulary we will go over um you know create a new vocabulary to handle um PWD okay so PWD or the variations could be PWD psswd you know things like that right uh now synonym is what is the English word in uh from the dictionary that we want in nlu to understand this and here now there is a cool trick to convert you know these multiple variations into our Edge X so you can just click pattern over here and then the system uh converted this into a regex for us but it did clear out the synonym values so we'll add that again and then add you know and we can do similarly we can do uh for application as well and then once the model is trained uh and then we test it it will understand the term pwb right uh next we'll go over some incorrect grammar capitalizations so for that I'll go back to my intents and I'll open um reset password again yeah so here the very third sample we have can I get a new key pass key so this the I if we can see it's uh or maybe it's hard to see but it's not capitalized uh it needs to be capitalized you know so uh we need to be mindful of such grammar issues right um I have one more example of incorrect grammar in Outlook desktop setup so here uh you know we have the word setup uh you know used being used over here now set out uh the word without a space uh it's a noun but the verb here we are using it as a verb so you know it needs to have set space up that's where it's a verb you know so little nuances like this uh if we are careful about it can help in the model performance so it needs to be you know uh entered like that okay how to set up my email application all right next we'll look at some typo examples um in desktop setup itself I have um some type of I'm trying to remember and trying to look for it maybe it's not in this one um it's in Outlook General issue so yeah let's maybe like stick to just one or two intense and just fix all the errors that are going through thank you yep yep you're almost uh getting yeah so here I can't send any emails right so can't is not an old word right so you need to have cannot or k n uh CA an apostrophe t Okay uh next week let's look at uh low context utterances so uh again like Victor said I'll just go go over one example you know and then in the interest of time uh Outlook General issues has uh low context utterance um we will go over [Music] so here like see if we see this sample Outlook uh you know what does that even mean right I mean you know it can be Outlook bot right so we need to add more context to it uh you know I need help with Outlook I am having an Outlook issue you know things like that okay uh next we look at ambiguous utterances which is uh similar to low context utterances uh and uh open reset Citrix session so here again you know we have a sample like reset uh Citrix session you know there is not enough context that is and it's pretty uh vague you know what we are trying to say over here so such samples are not uh advisable uh next we look at foreign language samples you know in the um intents you know having only the samples for the um current language so as we can see we have a Spanish sample over here and research Citrix session you know it needs to be um translated to English and then we can review our samples over here and see if we need to include that or we can get rid of it okay so next we look at model structure uh in terms of minimum train samples uh and uh disproportionate samples so as we can see clearly you know we have this reset password we only have eight samples for this one and workbench is telling us it needs more utterances that we need to have at least 15 samples you know for this uh intent to perform um uh and then similarly you know having disproportionate samples you know so as we can see uh almost three or four of our intents have are in the range of 15 to 20 21 uh utterances per intent but then if we look at reset Citrix session and RSA token they are like over 30 right so you want to look at you know um both these intents to see if you know there can there is any redundancy or is there enough coverage uh and then you know remove them or also look at the other intents to you know kind of have more samples in those you know otherwise it will give undue advantage to one end time one that has more samples over the other ones right and they don't have to be exactly the same but you know in the ballpark up about like five or ten uh you know within that range difference okay so next we'll look at overlapping intents uh so as we can see just look by looking at our intents over here we have like four Outlook related intents in our model Outlook uh General issues Outlook desktop setup phone setup and Outlook licensing so what will happen is you know if your uh your the end users are requesting Outlook related issues uh sometimes there can be you know overlap in uh the intents and it might end up the system might end up showing that you know more than one intent match you know and then that can get confusing so so instead like we said like if we consolidate all these Outlook intents into a single intent and then handle the variations via entities then that would be helpful so you know over here I have uh we already have a good model uh in place so I'll go ahead and review we'll go ahead and review that model so here um this model is you know with the corrected issues within the model so as we see we have only one Outlook issues intent so what we did is taking the Outlook General issues you know that's basically that intent and we renamed it to Outlook issues and within that we brought over the samples from the um other three intents uh into the model so this uh and I'm sorry into the intent so this intent is now handling uh mobile phones set up desktop setup licensing and Outlook General issues in addition to that we added like uh two additional entities and you know you can look at the need and add more as you go but if you look at Outlook issue type entity uh it is handling uh you know the different variations like receiving email sending them these are all the general issues but then it's also handling license issue and desktop and phone setup you know so that way if uh you know uh within your flow within your virtual agent flow if an end user is asking uh you know for a specific uh issue then you know it will have an entity match and that question will get skipped okay so that's how they uh can consolidate uh intents uh and create entities next we'll go with um vocabulary uh you know uh vocabulary sources let's let's do vocabulary sources also here in the start I already have vocabulary sources are open and we have um table vocabulary source and list vocabulary source so we can look at uh table vocabulary Source um I thought it would be a good uh idea to just go over the example that we talked about as a No-No uh having two different vocabulary source with similar uh columns uh you know that have two separate synonym values so here we have user vocabulary Source uh the synonym is employee okay and then we have um terminated user vocabulary source and its synonymous terminated employee so this is a No-No you know so you you don't really need this terminated user uh vocabulary Source because all you're trying to do is matching uh the user's name uh to using this vocabulary Source right so you know as long as you have this user vocabulary source that should suffice um another example for list vocabulary sources is having uh you know two different list vocabulary source with the same synonym you know so here we have a collaboration functionality um and if you look at the properties uh it the its synonymous function okay uh now we have another uh vocabulary list vocabulary Source in this instance uh called math function and that also has its synonym as function so then that can also end up creating confusion right because a collaboration function also means function and math function also means function right so you can correct that by setting this synonym as arithmetic function something like arithmetic function and then setting the other one also accordingly okay so yeah that that wraps up our um you know exercise sessions so I think we should save some room for uh q a um and I'll stop sharing awesome um I'll go ahead and just share our um q a slide give me just one second here uh skip skip okay um so yeah so we have a couple questions and I wanted that I wanted to uh go over live and um the first one is what from Edson what is the best practice recommended for transitioning from keyword only to hybrid search like keyword plus nlu uh would be possible to make it in ways like the first second uh per 10 so on so um so it's not possible to use both or let me let me put this way it you have to toggle either or keyword or nlu you can't kind of do a blended approach that means uh you know that that being said what you what you can do is um let me think about this what you what you probably have to do is um while your production environment may be using keywords in your sub prod or in your pilot instance you may want to like you say take the first second ten percent topics and convert them to nlu and test them like uh what we showed what nilima just showed make sure that um your test utterances accurately uh predict the intent that you want and then when it does you know Bank them that's those are good topics and then when you get to a good amount of topics that have a good nlu performance um you can switch over that production environment from keyword to go from keyword to nlu with those list of topics now some of your untested topics May um not be is out and and to convert them uh you know fully into production yeah thank you Victor and uh also to add to that what you could do is you know depending on uh what your intents are you know have like uh you can look at your uh logs like the interaction on the interaction side to see you know what uh end users are requesting uh you know for those intents and then collect a test set you know once you're building your nlu model to have a test set that is labeled and you can you know kind of do some testing you know initially to understand you know how uh your analy is performing you know what's this keyboard yeah so then the next question we have and you know keep your questions coming uh when the Spanish word intento is put into then sorry for the pronunciation but when the Spanish word is put into an utterance well the system also detect uh the word without the accent above the o or where we need vocabulary for that do you know uh yeah so uh basically uh so you're saying the question is basically assuming that it's a Spanish model uh and uh you know you have that is basically do you need that uh uh additional abbreviation like the expression on uh with that right I don't believe so yeah I think it should work even but uh in Dental yeah oh so you don't need it huh yeah yeah the best thing would be to options thing is to actually verify uh just set up a test and run a test and and you can verify whether or not it's actually working as expected if for some reason it it should but if for some reason it doesn't then you could easily set up um a uh a vocabulary for that um it would mainly happen maybe if if without the accent if it actually had a different word or in a different meeting so but you can easily test it yeah yeah that's a good point um if if if a lack of expression makes the word completely different then yes you're going to definitely want to make a differentiation like you know the rule of thumb as niloma uh has presented is you want the training data to be as accurate as possible you know users are going to type whatever but the more accurate the training data is the more likely you know our nlu will pick up on the things that your users type and be able to more accurately predict or predict with a higher confidence value and that's what you're trying to get at basically um cool so next question Edson asks uh what is the recommendation related to the governance of this practice there are recommended groups or roles to include members like a publisher approval something like that to focus on managing a large team yeah that's a good question Edson for for my part you know we recognize that you know for our customers a virtual agent or nlu uh implementation you're we're not going to have large teams you'll probably get like one maybe two product owners and this may even be a part-time you know assignment for them at that but we do recommend at least you know someone who is knowledgeable about nlu to you know create the training data I'm sorry create the utterances as an intense right and the the the the types of intents the type of utterances you're going to get your input from your business press owners your business units right you take that information um the the product owner the let's say the virtual agent and your product owners they create the models hopefully not from scratch right we do provide a lot of out of box materials for you to build off of and then um you know that person is uh then responsible for uh training and in testing the model so forth um of course you're gonna hopefully be able to delegate again to Pilot users or business press owners like hey does this look right you know kick the tires here's a bit and that uh product owner well then to the nlu based on that feedback so uh you know you know borrowing having say services or a large team that's really the way we recommend these implementations for nlu go I don't know if you nilimer and have any other inputs to that yeah no that that sounds good so yeah next question uh yeah so Heather asked in the chat you know whether or not we should add 10 of utterances that are irrelevant to any of the tense uh yes for your test set so um I believe nilma has done in a previous Academy um how to do um uh what's it called feedback loop and the testing batch testing right right so definitely you want to include some some of that in there just so that you can see like what happens uh I I wouldn't create any dummy data in your training data because that's what's that's the one you want to be more accurate uh but definitely in test set it would be very helpful cool uh next question by Eric I use a vocabulary source for catalog items that's good I have a bilingual model English and French okay um my English vocabulary sources sync properly but the French one always fails to sink any clue on why I have this issue by bilingual model do we mean uh like two separate models within the um sorry Eric also just said further down ignore my question he just figured it out so yeah I think um what nilma was going to say is like there should technically be two models right a primary and a secondary right like the primary model is probably going to be your English and then your secondary model would be in your other language such as such as French and that should hopefully uh clear things up a bit awesome yeah it looks like Eric got his answer yeah cool so moving along Valerie asks would there be an in-depth article concerning map entities out there um we have one for I'll go to the community in fact oops not this article we have one for um vocabulary vocabulary sources we have one for good nlu models we don't have one for mapped entities though but if you have any questions about mapped entities uh let us know uh type in the chat and I don't know if nilma you have like a yeah I think that's a great input I think there's opportunity for us to create an article also we'll we'll work on that yeah map a different kind of mapped entities you know it could be the capital resource and all of that you know yeah and then uh last question next uh from Heather expert feedback loop will incorporate real world real world utterances into the training utterances set a veteran says in training should be clean and ideal then shouldn't the admin clean up the efl sourced utterances of the training set um yeah so this is basically you know the uh the capability of export feedback loop to add like you know end user sample uh that uh you know needs to go into the train sample so there uh there should be added capability so when you are adding those samples uh in the train set you know making sure that um they they adhere to all these guidelines you know so yeah and changing them if if they don't yeah so like if I if my memory serves not not every utterance in the feedback loop gets added to the training that's you have to like manually add that right so yeah at the good ones and then you know probably don't add don't add the bad ones but it's helpful to know what you're accustomed to our our users added or I'm sorry our users act in unpredictable ways sometimes so it's good to figure that out you know find discover that and and create a training set accordingly uh and then again use a test set to see uh you know if some of those variations are accounted for or or what happens once when users try to type something like that uh yeah and so yeah the Heather is right I mean she just posted a comment that yeah the real world princess will be added uh you know to train and test utterances yeah we're curious to see what the experience was for that you know yeah Heather types in the chat efl expert feedback website you have to try to figure out what yeah first of all we'll add robot utterances to both the training s and test utterances interesting yeah it's I I want to ask Heather if she's seen like you know uh a princess that shouldn't have been added to the model getting added you know that's a good uh Point yeah you know it's okay yeah we'll take a look at that header because you know yeah we'll if if that's a product thing then we'll take a look but um cool well we're we're at time thank you everyone for joining thank you everyone for sticking around I know this is longer than the usual 30 minutes although you know we felt that this was important um like I said the academy will I'm sorry the academy recording will uh will will be posted soon our next academy uh will be in two weeks and it will be uh about uh routing uh with uh to eight with agent chat right into separate cues or separate agents uh that's on January 24th which is a two weeks two Tuesdays from now uh thank you everyone for joining and we'll see on the other side thank you thank you