all right good uh thanks everyone for joining the session today uh we are going to touch base uh how what are the best practices that we need to follow or we can follow to implement our uh natural language understanding uh that is one of the offering uh within service now platform uh before we jump in um quick reminder on the safe how notice um I'm going to U use statement uh if any statement is forward looking then uh we don't want you to um you know make your buying decisions based on those forward looking statements so with this uh let me move on to next slide uh little introduction about me uh jitu Johan based out of service now uh India Bangalore office uh I'm part of product success Management Group here and uh uh really welcoming everyone um hopefully you will have a uh you know better understanding on the nlu best practices today um with this um so similar the one we have live now session today we do have few more session lined up um something that you guys can also join uh in future uh let me put the link here so you can follow this link and and please do sign up uh for future sessions all right um just quick h rules so this session is is going to uh record and it is going to post it on the community um site service now community site so uh you can access the those recordings uh you will also have uh the PPD as well there um um during the mid session if you have any questions please do utilize the Q&A button available in in uh in the Zoom app um I'll try to pick in questions in between um yeah and don't forget to fill in survey so once we complete the session today you will have a a survey prompted uh please fill in your responses that is those are really avilable for us to uh improve further with this I think we can start the session before starting the session I would like to know uh about the audience um I will just run through one quick poll and I would like everyone to respond that poll just to see um how much aware uh you are about the nlu um so maybe some of the folks they already using nlu then it's good like all right give me a few more seconds and let me switch off my video to have bigger screen good I can few more just waiting awesome awesome I think I can now and the poll I think two folks yet to respond all right good let me end the poll good uh so what it looks like uh we have majority of the folks we have they are already using or they are familiar about the nlu so they're looking for Best Practices tips and tricks uh few folks they're not familiar no worries I'll just try to cover some basics in few five minutes and then we'll move on that best practices awesome good um let me close this um all right so uh before we jump in um I would like to just let you know a couple of terms uh which are generally we use in our day-to-day human conversations so there is uh one of the important term called intent hey before going so why I'm showing this picture here is um sometimes you know we have um our own understanding about keywords and um and the phrases that we use so if you can see here I can recall one incident where um you know um few folks in the team meeting they they were talking about uh um hey where is my zebra where is my zebra so so one of the contractor guy he was really shocked you know why you guys are looking for zebra so what their intent was basically their intent was to uh calling zebra is uh zra printer um so there's company who make uh uh printers the company name is zebra so you know so this is something sometimes it's very unusual U uh we as a human we will try to draw intent uh which was not uh you know um which is not assumed by the uh by the speaker so uh this is where nlu comes into picture uh nlu is really um able to help machine to draw the intents behind those conversations so if you can really train your model then literally system can able to differentiate U uh based on uh based on the samples you know people they are talking about uh the zebra animal or Zebra printer so uh probably we'll learn more about this uh about nlu before we jump in quick uh reminder on couple of uh important terms so what we call intents uh intents is something that user want to do or user want to perform uh kind of action so if you remember uh if I can just give you example of uh I want to update my email ID right so what is the intent behind it update so that's what the intent is and these are the very important terms I'm keep using during my session um so intent is nothing but um what exactly user want to do right um utterances are the way user will uh uh will speak the way user will uh communicate uh about the intent that is what the utterance is uh if you see example um you know incident sort description um if you see the S description people some I'm having issues with the with the email ID so uh and they can use different phrases uh my email ID is not working my email is not working I'm not able to send email so those are different ways that human can uh put together their description of the problem so that's what the utterance is uh entities is nothing but the object or the context uh that we are talking so if anyone is is saying that hey uh my laptop is broken so laptop is nothing but the entity uh in this uh uh address yeah uh we'll talk about the different type of entities we have three different type of entities um defined system entities which is defined by System user can also Define their own entities um and we have some common entities um example currency or date all those things there is important concept called vocabulary so if you recall the example of zebra here um you know um sometimes people they use same terms in different uh ways so what we can do um and even sometimes you know words they are not part of dictionary um you know acronyms uh some indust stand terms um or terms which are used you know in different way uh zbra for animal as well as printer so you can Define vocabulary um and really you can indicate your system at what point of time system need to differentiate uh you know what is the correct synonym for any term so if anyone you know um typing a zebra then system can understand yeah so usually um in my organization people they talk about zebra that means printer not or the animal so you can Define vocabulary um we have something concept called nid models so those are the machine learning models uh you know which need to be trained on Sample atanes and they will help you to predict the right uh uh uh intents behind any atance so these are the basic terms before I move on let me know if you have any questions okay I don't see any question awesome uh all right so um just a quick uh um a quick summary how how that exactly works in the background so if you recall um one of the request from user say um you know uh that user is interacting with the virtual agent and and want to update email ID right so in the traditional way um in the historical way we used to have something called topic uh uh conversation so uh before nlu world before machine learning world we used to have this kind of flow in the system which will um start uh once user type any atance or question you know it will go step by step uh okay you want to update so virtual agent may ask couple of questions just to confirm because that is not smart enough right to to determine the intent behind the actual uh users question so you know we will ask question he do you want to update the email ID yes then user will say yes then um we again will say okay what is your uh new ID that you want to update so all those are additional uh steps were there which was really timeconsuming and not not so smart um enough so nlu is really U making lot of difference there uh nlu is smart enough to decide to pick the right intent behind of any uh atance so if any user want to update email ID then it can really break out the entire uh phrase into different part it can really determine the intent behind this uh conversation so intent here is update profile um it can really determine what is the item here so item here in this case user want to update the email ID right and what is the value for this this item is value so basically nlu can bypass lot of unnecessary questions and make your conversation smart to update or you know act on the user request uh by making some decisions based on the the models or the predictions right so you can see here um with help of nlu system was able to update the profile um what what field in the profile basically email ID and uh update the new email ID without asking any more question from user so you can you can imagine how smart uh you know nlu or the the model world is okay good um let me move on further um so since we are in the uh we in the best practices session today not the implementation of nlu so I'm going to skip the implementation model uh implementation cycle I'm going to talk about the tuning cycle here uh before that I can see one question item means entity you're right yes yes ra so there's a question um item means what item means entity yes so in previous example email was item or entity yeah okay so I was talking about the tuning cycle right so instead of talking about implementation cycle I would like to focus on the tuning cycle oops close all right so um so this is out the cycle box um typically um we plan our um nlu models we build those en models in service now and I'll tell you um how easy it is by using the workbench um and then we'll test the test our models um you know uh um if if we are doing this first time then probably we don't need to tune so it is aune then um you know if if results test results are satisfied enough then go and deploy monitor uh with time move uh there you know U there is a recommend that you keep eyes on your model per performance and uh time to time you need to tune your model so that you can have a best performance as you expected so this is what the cycle you can follow um once once you think that right time to tune and how you can determine the tuning you can you can use couple of tools uh within service now to check how your model is performing so if you want to follow the tuning cycle you go back and uh make the changes in a model and test again and then you know uh tune again until it is reaching to the acceptable quality um and then again Deploy on production uh keep eyes uh over your model performance um and then again Follow The Improv Loop here if you see some drift in the model performance so again test cycle so this is what the typical cycle that we recommend to follow uh to tune your to keep your model up to the mark yeah uh before going further I would like to just let you know the Fe uh how it looks like basically um if I can show my instance here so I am here employee Center in my service now instance uh if you can just quickly um give you example how it how smart uh nlu can be so I'm a user I'm interacting with with my virtual agent and this virtual agent basically is connected with the nlu models so let's say if I want to know what is n ldu yeah so you'll see uh system will start responding uh you know because it determine the the intent behind uh behind this right so basically user want to know about uh nlu or natural language understanding even smart enough to uh differentiate energy versus natural language studing so you can see the responses from uh VA uh you know instead of asking multiple question directly I have results with me and no more so there is a there is a virtual agent flow behind it which is utilizing the capability of nlu to identify the intent to uh you know uh make uh more relevant responses to the user yeah so this is about uh the very quick uh basic uh you know example of how capable nlu is basically all right so let me go back to my presentation and I can see there are a few questions already already done okay how much testing tuning should we do before we deploy should aim okay I'll come back to this question on the train tuning what is the source of information for what is nlu I will also explain this source of uh uh nlu I will show you the flow very basic flow uh there's a question it is taking details from Google about NL know you you can of course it's up to you you can configure uh the background um of the responses so responses can static be responses can be from um you know service on instance so you can configure those uh uh things in the background background in the virtual agent flow all right right um okay so let me continue here um there are few questions I'm keeping on hold for now okay so we we touch Bas on the um anal un cycle and before going the bestes there were there were a few questions um what is the how we can you know um tune the model um and those things so I would like to just give you the best practices around uh model planning um so you can utilize the workbench within service now um to create your model to um use the auto the Box model that we already Supply with your instance uh and of course if those are not sufficient you can always create new models you can uh tune your models uh as per your need so um so always recommend to you start with out of the box models that we Supply with your instance I use topics intents for actionable Content so um if you um you know if you recall the basics of um workbench and you need to define the intents and erences um in case people they are not aware I would like to just quickly show you how the workbench look like um yeah so if you go to instance search for nlu um then you will you will see the the anal workbench here and this is how the workbench look like so I clicked on models and if you see I do have couple of models already drafted you can see couple of models uh you know uh pre-up with your instance so you can utilize those models uh as a starting point then you can always uh create your own models so if I take example it's very easy in the workbench uh you can create model from blank or you can import uh from CSP um you can use pre-built model as well so if I just quickly show you the model which I be using nlu right this is what the nlu model uh behind the VA conversation yeah so you can see here I defined one intent here um couple of entities um so not going to detail um because this session is for Best Practices not to implement the models uh but if you see here um this is what the intent uh which really picked by VA when I tapped what is nlu in the virtual and chat box um VA was able to determine what is the intent behind the the question so this what the intent was and how it was how n was helping so NL was able to help VA uh by having those uh sample trenes and uh this particular model trained on these sample trenes right so this is what the the background brain behind nlu and someone ask about the flow um I think if you go to your virtual agent conversations uh conversation interfaces uh go to Virtual agent settings you have to enable this option to use start using the nlu and of course you can configure uh different nlu providers by default it comes with service now um nlu provider but you can always configure uh more if you have you can configure the IBM Watson or Google uh or Microsoft uh anal providers as well so yeah so this is what the settings and if I show you the quick flow which I was using so you can see this simple um flow in the virtual agent designer um you can see this is what the flow configured and based on this flow I was getting the answers or relevant uh responses uh the interesting part is um whenever you want to use nlu in your virtual agent flow uh always you need to go to properties and um select the uh the topic U sorry um this one here nlu model so you remember I have one nlu model already preconfigured um so this is how I can associate my model I do have all the other models as well so I want to use ANL model here to help VA to to you know utilize nlu capabilities during conversation and I I can also Define the intents so if one model is having different intents then I can select what is the intent I want to you know VA U predict for this particular conversation all right so the moment you configure the nlu then you will have this kind of page enabled in your designer via designer right so you can seees and entities uh everything here all right I hope this answer a couple of questions let me go back uh to my PPD talk about the best practices so yeah we were talking about the out of the box models right always use topics and intents um for to Define your external contents um uh fill knowledge gaps by building needed search content um um try to avoid creating intents that can be easily solved by um AI or there is offering AI search within service now um uh uh instance so you can you can always leverage those AI things for um you know instead of creating model for those uh intents those can be self solved right uh always recommended to use vocabulary items um and sources that we already have within service now I'll tell you the I'll show you the vocabulary items and source is always test and tun your model at every stage so U whenever you make any changes always you know test the model and uh tune according to the expectations and of course you need to the recommended guide guide RS which I will show you in few minutes uh from now um then there are a couple of best practices uh and in the model structure and size before I jump in there quick check on the questions um so there was a question I think already answered how much testing and tuning we we do before we deploy so in couple of uh minutes I'll show you how we you can best utilize the available tools to you know to make sure that your models performing well and uh responding of course you cannot cover 100% uh coverage but you can optimize your model always um so this one done then question will there be any session about NL implementations um there was one session few months back um probably you can just go to um Community or uh YouTube you will have an implementation as well session as well all right good um there's no question can multiple models be selected for VA or only one um no so as of now when you are selecting your conversation you have to select only one model um even one intent you cannot select multiple intents um if you select multiple intents and then your uh uh your you know V will will confuse and start producing wrong results so you need to make sure that you are segregating your intents um while building the model good is there any documentation for according going to the differences of when to use AI search versus nlu um I think I can help I can attach uh um the community links along with this presentation so so once you have this presentation on the Community page you you can refer U the links uh over there to differentiate when to use AI search and uh what is the what are the best um scenarios to go for NL all right good um so there are some best practices which I would like to highlight uh uh on the model structure and size so avoid creating models that have twoo less intents um uh for example less than five intents so if you have a model less than five intents then you not able to achieve the uh the right prediction um results U you know you will not cover the entire um uh and so basically you lose the coverage you're not able to cover the the um the area that you want to Target for Ann prodution so try to U make intents more than five uh avoid creating intents that have two lesses um less than 15 so at least you need to provide uh minimum lb of samples to train your model um um so you know so that your model can start predicting um almost like a good amount of uh scenarios yeah avoid creating models that have very disperate in size to balance the strength of so this is what something um um so if you have um say three models or four models if you have five different intents or 10 intents uh try to have some balancing um with the training samples so try to make a similar number of AES or similar number of samples against those intents so that you you your models uh you know they will um they will have um a level of uh matching right so so imagine that one model is having a five um intents each int having some 10 erences another model is having a 200 or say 100 intents and each intents having 500es so it's it's not a a great balancing out your Model start predicting the wrong uh intents yeah always try to refine um training atanes so so definitely model need enough training samples to balance a quantity versus quality so as I mentioned right if you have a two less samples then you're not able to um really provide the full coverage of your models uh don't add near duplicate samples so uh just because you need to provide a minimum samples don't try to make a duplicate uh samples yeah so for example um I need a laptop I want a laptop right so try to avoid using those kind of uh samples or erences while training the model basically they are kind of a same thing right so avoid Source duplicates don't assume model can handle implicit statements like human brain um so this is what something avoid using a trenes or samples for example like uh um my laptop is broken right so system is will not able to differentiate since your laptop is broken broken you want to order a new laptop but if you talk about you know to talk to any human then human can determine the next action right so try to avoid those Simplicity statements um system you know models are not smart enough to uh differentiate the implicit statments like human brain right if you really want to order laptop then you need to provide a sample uh like a system is broken then or the laptop so you define the actions behind it all right with this uh before I moving further I do have demo planned so probably I'll just show you all these things uh in my system as well before going to the next slide which is more about testing and tuning a model so there are a couple of important tools and techniques are available supplied within your inst you can use those tools to uh to you know tune your model and get the best performance to check the performance of your model all right there's a question uh oh it's old question what is the source of information for what is nlu question okay so source of uh question was from R and answer of this question is um the source is as I explained ear right you need to attach your model with VA conversation then VA conversion will start interacting those intents with the help of nlu models all right good um so if no more questions then I can keep moving it um just to let you know there are couple of important tools and techniques available within service now um can be used those techniques and tools to keep eye on your model performance to tune your model to check how how your model is doing so um first one is about you can utilize the workbench um to compare uh the draft model and published version of your model so if you have one model model which is published on production uh you made some changes in the model uh now you don't know you know um how it is going to perform before publishing so if you want to know before publishing you can use this compare option I'll show you how you can utilize this um nlu expert feedback so um this is really a good option where um system will give you um option to um you know you can sit down with your expert within your team to to review couple of actual atanes happened on your VA logs so system can pull those logs and it can offer you to review those conversations and see if if you have right predictions for them or not if you want to make any changes you can really give your feedback uh to improve the model performance for those examples multimodel best testing is really a good uh tool you can utilize uh you can really um test your model uh against a large set of uh samples you can evaluate the performance of the model against few of your sample data right it's very easy uh in the system uh there is a to called Model optimize feature so um here is something um system itself will determine uh and it will optimize your model and it will enhance the performance um by using the incorrect predictions and improving the correct prediction so these are the tools available within service now you can always utilize them cross model conflict review is another tool which which can really identify um conflicting intents within your models so that you can take uh corrective actions you know you can uh you can change those atanes uh where you have a conflicted statement or conflicting examples so if you have conflicted uh conflicting statements or atanes your model will uh always confuse on those attes right what is the right intent so I'll show you the example um how you can uh use this nalu model performance for VA is something um you can see the performance how well your models prediction uh how it is how good it is predicting the intents uh by based on VA uh and user confirmation Now intent Discovery is something this tool will help you to dve more relevant intents which you can build in your model and uh how this will suggest it will suggest based on your own data uh which is available within service now instance um so it can help you to basically to Define uh new intents to avoid the deflection of uh incident or cases right so before I'm moving further the next uh I have demo uh yeah let me show you my instance so I'll try to explain how you can utilize those tools within service instance all right so you can always go to nlu and the first one was compare right if you want to compare um the results how your model is predicting or how your model is predicting post changes uh before deploying on the production right so the easiest method would be um if you can go go to go to models yeah so I can take example of one the model which uh can be so for example I want to check how this model is predicting so I I make some changes I have add a few more uh uh erences and I want to test how it is performing as compared to the my published version of production right so it will give you the results um uh if you type any atance then how it is going to perform on the current published version as well as the the draft ver right all right go here um and I can always use this try model so for example if you want to say I want to check uh I make some changes in the transes now I want to see how it is going to perform uh uh post draft compared to the my current published version so you can use this uh so you can see uh this is what the result um on the from the trade model and this is what the result from published model right right so I can see if it is really improving so I can see um on the current published model or the production model it is predicting around 94% data analytics and post changes uh there is slight Improvement in the production so seems like my changes are working fine uh they are really U going to improve the prediction so I can uh publish this model yeah so this is one way uh and always you can use uh one more method yeah you can you like something called uh you can use you can do the same thing um how your model performing U here by using this test conversation okay I'll show you this test information in a few minutes all right so this is about the um this is about the model compare uh what else the next I want to show you the nlu expert feedback so if you go to your nlu you will see this option called expert feedback loop um which which really gives you um you know option uh to provide your feedback on the actual VA chat locks yeah so if I can just quickly show you yeah so you can see uh it basically um system is going to pull the data from your VA log your actual production VA log and it is going to suggest couple of um uh atanes uh you know to share your feedback so I can see here uh example I can I think this one is good example so you can see here uh here I have some of the intents and these are the couple of utterances or samples so as an expert I can give my feedback whether those uh uh those are trenes uh really uh predicting the right entity here so email is not working it's predicting demo how B which I I think so it's a right prediction so I can give my feedback hey um this is mismatch or if it is predicting rightly then of course I can say it is matched um so it's kind of a you're you're giving a feedback to the model right so if I say mismatch then I do have option to correct it so what is the right intent um you know model should product for these kind of uh rances right so you it's a really nice option to um give feedback to your model and let your model improve uh further so it can improve by using this button here but unfortunately I don't have enough samples to utilize this but yeah so this is how the N expert feedback is I can see few questions um just take few more minutes to ask respond these questions all right next thing I'm talk about the multimodel best testing um so if I close this yeah so there is a tool called multimodel best testing you can basically so if you recall my statement right so multimodel best testing will give you option to uh test your model performance on large set of data right so this large set of data uh uh you can upload here your sample or testing data set so I have uploaded one data set here 19 and you can really um run and analyze you can um compare how your model is uh performing uh against this sample data right um so if I just quickly show you example uh I already run few testers so you can see the the report here it is clearly showing that uh 46% it is uh you know correctly predicting 33 33% where we have a multiple predictions um luckily we don't have any missed prediction um and 20% atanes where model was not able to predict or it basically predicted incorrect intent so you can check the performance of your model uh against a large set of sample data right you can also uh details here what are the incorrect Tes and per incorrection you can also see the detailed report which are the sample is uh correctly incorrectly uh predicted right so this is about the multimodel best testing very useful tool um to check the performance of your model um there is something called Model optimize feature um so you can actually um optimize your model and service now um instance is smart enough to optimize your model um which is uh which is nothing but uh enhance the performance of your model by reducing the incorrect predictions and improve the correct predictions right so how you can do this uh you can go to bat test and go to data set one or actually not so you can run the analyze here and when you click on analyze you have option to optimize your models so you can select this um you can select your data set against which you want to optimize uh you can select your model which model you want to optimize so system will U uh run the automation behind it and it will try to optimize your model uh you know by removing the in correct uh predictions and improving the uh model performance so if I can just quickly show you the example so I did run through one Optimizer already so you can see the the performance uh this is what your current model performing is this is what uh if you uh once you run the optimize then you can compare the results if you have a better results with this optimization then you can always uh update your uh production model with this new optimized uh changes right so it's a good tool to automatically enhance the performance of your fa model uh by using um this tool automatically you don't need to really run behind each and every Tres so system will will really improve the performance of your model and you can see here right uh which are the intents need uh need your uh and even you can see the so these are the changes right uh once you run the optimization these are the changes uh system uh did in the model to have a better uh performance so when someone says ticket class um the expect intent was book applied Uh current uh model is predicting this uh intent however post optimization it will start predicting the right intent book of FL so you can you can make a decision um if you want to go ahead with these optimization or not all right this is the optimization model cross model conflict there is in the concept called cross model conflict so cross model conflict will give you um a good uh um you know it will help you to identify the conflicting intents um across your models so if you run the uh analysis then you can really see um if you have any conflicting statement within your model so you can see I've already done one analysis I can see uh it is showing me I have two critical um uh instances or the arces where um my model may get confused right we have a conflicting erences so you can see this is what the critical is so you can see exactly um it is part of um one of the intent uh laptop issue so your um so basically showing atance so there is a one atance which is going to predict open it ticket however at the same time there's uh similar atance which is going to protect hardware issues so if any user uh type a laptop issues then model is going to confuse right will not perform because um it will not able to make a decision whether I should um open ID ticket or I should suggest the hardware issues so you can always um use this tool to have those to identify those conflicting uh statements and and correct them for uh so we have two different kind of conflict reviews one is the critical and moderates you can see modate also so I think this is really a good tool to uh identify the uh the statements which are which are really making your model performance down and of course you can ignore you can change uh you can correct them right so this is another Tool uh there is something called nlu Model performance for VA so this is something you can utilize this one all right so you can utilize this tool to to see the uh performance of your VA conversation so if I can show you um there was a set here so this one right if you enable this then what it is going to ask you it is going to ask take a feedback from user uh whenever any user type a question in the VA it is going to ask a it is going to ask the feedback from user whether I predicted the right uh prediction or Not So based on those responses um you will have the you know the data here so you can see here 20% of time user confirmed the prediction was correct right so so basically um you know you can you can see the how your model is performing uh on the actual VA conversations in front of user um luckily 0% time no confirmation made by user 80% of time user confirmed as incorrect so you can see how bad my model is performing right so there is a good amount of requirement to review my model to tune the model so that I can have uh you know better uh predictions uh in the VA conversation even you can see the the details as well uh which are the utterances uh predicted you know predicted intent and uh which model was predicting uh what was the outcome so menu was presented to user and users have not selected any response or any feedback right so you can see all these details as well and this is tool very useful for um for the actual performing your how your model is performing in front of VA now with this I would like to go ahead next toward the intent Discovery so there is something called intent Discovery available here you can utilize this let me close on these STS yeah so intent Discovery basically it give you uh opportunity um to discover new intents which can deflect the incident on on your instance it can um really fulfill the users request at VA level instead of uh creating incident to the user right so how you can do it system will dig out your data available on instance and it will start showing you the potential intent you can build in your model to avoid those uh uh incident right so if I just click on run analyze then um okay do it now so you can um you know which are the fields you want to analyze uh so you can select um for my example I have gone through the incident so give me the uh potential intents from the incident table right so you can run this report and how it will look like it will look like something like this so if I see this report then I can clearly see there are few intents which I can build in nlu and it can start answering the the questions raised by users uh in VA instead of going let them go and create incident right so I can create one intent around employee payroll so this is what the data you can use to create new intents and uh you know this is something coming from your instance your actual data so uh always trustworthy right so I can build one intent here to avoid at least 293 of incidents in future or I could have avoided 293 incident uh by building this intent within my VA conversation right so it can satisfy the answers um raised by user within VA instead of let user go and create an incident for this small uh or whatever problem he asking so this is what the intent Discovery is very useful tool all right before I go into the next section I would like to take few questions um any do expert feedback loop would you recommend having expert review locks once a week for the first month or so to refine the model something we have to periodically do um yes so I would recommend yes we we do need to periodically review because you always your instance will have a new data so um you know try to review it periodically and see if you have a good percentage of uh uh record showing for new intents always you can create new intent based on that yes it's recommended all right good now the question is can this feature be available in PD instance uh development instance uh I think developer instance I need to check if it is really available on PD instance uh because there is separate plugin for those advanc nlu features so if you go here uh you can see these are the advanced features so you need to install the plug-in for to you enable these instance on on your nlu so I need to check on the PD I'm not sure what the developer instance they're available or not all right there's a question multi batch performance test for NLC multi batch performance yeah that's that's what the question that's what the answer is I need to check if really it is available on PD uh or not all right there's another question apologies if I missed it but do you need performance analytics to run intent Discovery no uh not at all so uh you uh I think what you need is is data on your instance um to filter out so this data is coming from tables within service now instance so if you have WR um instance then of course you can drive the intents you can discover the intents uh from your sample from your uh table data all right good if no more question I'll just go forward let me go down all right so we covered this uh so you can also um review the workbench properties um though we have you know default uh settings available but you can always uh review and tweak based on your own requirements so for example maximum number ofces per intent right so U recommended is U less than 200 um you know so at least minimum five and less than 300 so we put together 200 as a default uh how you can review these settings uh system properties you can review or um or I'll show you how you can review so there is a one settings option within nlu you can you can review those settings and and tweak according to your scenarios so you can see a lot of settings are available I'm not going to um highlight everything but you can see uh there is a conflict detection so if you're using conflict detection um tool then um these settings might be important for you um you know you can set that threshold for U critical conflicts and modor threshold uh you can also uh maximum number of records for intent Discovery classification right now minimum is 10,000 uh maximum is 300K so minimum we need at least 10,000 ,000 records uh to use the uh intent Discovery right so you can change but 10,000 something recommended similarly you can review other um settings as well U the maximum number of rows in a best test uh you can restrict the the number here uh and how you can review these settings so you can ask your uh admin to go the instance and either you can go here type nlu go to settings you can see those settings here yeah so ask your admin to um keep reviewing those settings and you know keep the numbers which are relevant for your uh use cases all right good uh let me go back all right so this is how you can review this properties um we already covered one demo so I'm going to skip this I would like to talk about couple of gu rails which are really important um you know kind of a best practices so don't use unknown words in your sample lances um for example like I am so I and space am is the correct word don't use um I am or always use vocabulary for your um acronyms um or your industry terms um don't use the the sentences or utterances which are creating ambiguities so um you can see the Outlook login Outlook meeting they are kind of very overlapping intents right don't create those intents uh low context rances don't use thees which are single word or not making sense or you cannot determine the context for example using the reset is it's not giving me anything what do you want to reset right uh don't use combined erences uh try to make it separate so I need a new laptop is a separate atance and I need a printer is a separate instance right there is a recommended intent structure as well um and you can follow at least five intents as we spoke earlier 15 samples per intent minimum uh so that you can have enough coverage and you can avoid duplicates uh as well this quickly move on always use vocabulary whenever wherever you think we have two different sources of the different type of w so you can Define the um vocabulary um you know um so these are the cases where you can use the vocabulary you can handle acronyms you can Define unknown words you can Define the industry specific words um in the vocabulary and then system will start you know referencing those synonyms whenever we saw the term um I'm going to skip this types um I want to show you basically I want to show you uh okay go to go to models here you go so I created one bad model basically which is having U wrong rances or not recommended I want to show you that model how it looks like so that you can avoid uh those atanes in your production model so if I show you email issues all right so these are the couple of ATT you can see this is my English model and I'm using using the the different language statement right which we we should avoid it um we can avoid one word statements or no context statement because we cannot drive anything out of these words um don't use um um non- English grammar things like I can't I can't is not a correct word cannot or apostrophe T so you can avoid those kind of attes in your uh models to have a better performance um what else uh I do have a couple of so you can avoid these kind of things I am it should be I space am right uh what else I can show you here okay yeah I can sendil attachment so you can avoid those those kind of atanes to have a better performance uh from your model yeah all right um good so with this I think we are done with the session today hope you learn um best practices I would like to have a quick feedback before wrapping up the uh session so I would like [Music] to raise [Music] a feedback uh session so I I hope you learn something new then I would like to know if you are going to try these best practices on on your instance right uh in the meantime let me see if I can answer questions oh this was good awesome thanks for feedback all right good good don't forget to share feedback U post session also try to join our future webinar or live now sessions to learn more about service now offerings