best practices when we're creating a pi classification model then we'll talk through performance and tuning of your model the different ways of deploying your predictive intelligence model and then we'll wrap it up with additional learning resources now you are welcome to follow along uh in your instance and if you want to get a jump start on that uh you go you would go to your instance open up your instance go to all predictive intelligence classification and solution definition so that's if you'd like to follow along or you could just sit back relax and watch and kind of think about what we're discussing today so i built uh i built this uh this content to help customers who started using predictive intelligence and predictive intelligence if you've gone into it it's very simple to use but as you go get deeper into it you're probably starting you're probably starting to stumble along some terms where you're kind of scratching your head going well what does this mean so some of the questions that you might see in our documentation or in the predictive intelligence interface are things such as you know artificial intelligence and machine learning and you'll see those used interchangeably and so some of the questions i get is artificial intelligence and machine learning the same thing or you'll see you know supervised and unsupervised machine learning so what is that and what's the difference right away when you're creating a model you'll see what is a a word corpus so what is that as you build out your classification model to predict things such as assignment group or category or priority you have to select inputs and so i often get asked so what inputs should i select and then when your model is trained how do you know if it's good enough and then you'll see these different statistics such as precision and recall and what are those you also see something like a classification threshold so what does that mean and then finally when you get through all that you have multiple different ways to implement your pi model and so we're going to cover all these kind of questions and get you comfortable with these basics basic ai concepts used in predictive intelligence so let's start with the first question which is what is machine learning and as i mentioned someone might say ai or machine learning in the same breath they'll use it interchangeably but ai is actually this broader domain that encompasses machine learning and so it encompasses machine learning encompasses deep learning in something called natural language processing and so ai covers things in this broader realm such as computer vision or voice recognition so if you're using alexa or siri that would be ai it also covers things where we give computers the ability to paint art create music or write code machine learning itself though is a subset of artificial intelligence and this is where we we use it to program computers to learn and identify patterns and this is where predictive intelligence falls in so predictive intelligence and servicenow is actually categorized as machine learning we also have deep learning in the platform which is where we use neural nets and it's called deep learning because there's multiple different processing layers to take an input and then match it and transform it to a specific output and then we have something called natural language processing which falls at the intersection of machine learning and deep learning so if you're using virtual agent you're using our natural language processing capabilities that you're using you're in that domain you'll also see in our documentation supervised unsupervised and so classification and regression which are two of the predictive intelligence algorithms are considered supervised machine learning algorithms and all that means is that we teach the model we train it on historical data with all the past answers and so data scientists would say data with answers we call that label data and so for example if i go back to predicting the assignment group we're trying to predict the assignment group all your incidents or all your cases should have a assignment group already assigned to it and we're using that to feed into the training uh data set of the model so that that model knows what the answers are so when it sees new things come in it will look back at the historical data look at the past answers and say i think it's going to be this i think the issue when i see you know i'm having an issue with outlook i think that's a software issue and that should go to the software assignment group unsupervised machine learning is where we find patterns without the answer so we don't have labeled data in that data set and so our clustering falls under here and so we're the unsupervised machine learning models by itself learns and finds patterns uh and um and surfaces it back to you and so we use that unsupervised machine learning that clustering capability into in our in some of our solutions such as automation discovery or clustering to find automation opportunities finally neural nets we talked about how it's a deep learning uh solution uh that uses both supervised and unsupervised um capabilities and that is also within predictive intelligence now servicenow uses ai to improve the experience within the platform the work experience in the platform so if i read left to right these are the four core use cases that we focus on so starting from the left we use machine learning to discover automation opportunities in your data so we'll use we'll use ai we'll take a look at your incident data your case data your request data your chat data and we'll recommend we'll use the ai to find hey we found an automation opportunity we found maybe a lot of uh hardware issues and we recommend that you can use this virtual agent topic to troubleshoot hardware issues or we'll use the ai to mine your your processes so we can mine your process data to identify bottlenecks when you're moving from different states so it going from open to pending to close to to resolve to close we can mine that and identify bottlenecks and apply all kinds of root analysis root cause analysis in your data the second area that we use ai to improve that experience is we use ai to improve the experience over your incident and case life cycle so things like using the ai to route work to the right place and also automate decision logic so figuring out like i'm putting in a change request and is this change request based on all these different features is it a change risk we also use the ai here to help your agents solve their issues faster and so i put on the bottom some of the different frameworks that we use to do that to accomplish that classification similarity regression are a few the next one is where we're using ai to predict issues that can cause service outages and so we try to get ahead of service outages by taking a look at your alert and log data that and then running correlation analysis and pattern analysis to determine if there are certain alerts or logs from say your splunk logs or from itom that that may attribute to a downtime on a service maybe it causes an outage on an oracle database causes a server to go down and so that falls under our predictive ai ops and then finally we are able to digitize documents now so when we combine it with our rpa our new rpa that came out in march robotic process automation with our new document intelligence capabilities which allows us to read through documents we can process through uh stacks of digital documents and pull that information out automate that processing and then feed that data into downstream workflows all right so i think i mentioned before ai is an in-platform capability the ai capabilities are classification similarity clustering regression and natural language understanding in our search we feed those we use those capabilities into our ai powered solutions so you'll see that if you're using agent workspace there's an agent assist capability which allows you to help agents solve problems faster so that that's using those ai capabilities i talked a little bit about process mining well that's in our process optimization the virtual agent uses the nlu so we have a host of ai powered solutions that use all those ai foundational capabilities and all those capabilities can also be embedded into your workflows to make them more intelligent okay and then what's nice is we have the whole infrastructure to take care of all the training and all the prediction for you all right so let's let's talk and get back to classification which is really the the focus of today's conversation and where we want to where i want to dive in a little bit deeper so classification in predictive intelligence is is what we call a multi-class classification algorithm so we look at incidents for example and we look at their key features such as short description location ci and we're able to predict an output such as what assignment group would this belong to so if i feed the algorithm a short description does it fall in the database if i look on my right does it fall into the database category security or network okay and so we have a number of these models classification models out of the box and we use those to predict assignment group category priority or change risk so those are some examples and underneath the covers predictive intelligence classification is using a default algorithm called logistic regression to sort of make these divides in this uh in this visual that i've got on the right here right so it's it's using logistic regression to do that and then it also has another algorithm called xg boost which is an advanced capability and very helpful if you've got skewed data and i'll get into that okay so this is this is what we're going to talk about and just to put a picture on how this works in your environment i'm going to show you an agent workspace an example so here an agent workspace where i can see all of my information as an agent all my information around incidents problems changes catalog tasks i'll go ahead and i'll create a new incident i've got a mandatory field so i'll call i'll pick a caller abraham lincoln and so you see i've got like a blank i don't have anything in assignment group i don't have anything in category i have a default category i'm just going to say 11g is out of space okay so i may be a new person right out of college um and i don't know like what is 11 g is out of space i have no idea what that is and so i'm going to use the machine learning the ai and servicenow and predictive intelligence to predict the assignment group predict the category but i could have it predict all these different fields and all these different fields are important because we use these different fields to route this incident to the right place so if i'm level one at a help desk and i can't solve this i need to figure out who i'm going to send it to right and so i'm going to trigger this off to save there's multiple different ways to trigger it you could trigger it off the tab you could just have a recommendation pop up but for simplicity i'll show you what it looks like when you figure it off to save and you can see that i predicted the database for the assignment group and predicted database for the category and it's gone ahead and it's changed the default category and it's changed the the assignment group to database and then i could go ahead and save it it's already saved so now it can be routed now before i did that maybe i wanted to tackle this and see if i could solve it so if i go over to this agent assist wherever it says similar similar these are also machine learning algorithms so i could say similar resolved incidents and before i even route this it'll go out and it will search for similar result incidents that look like the problem that i'm trying to solve and so i see 11g is out of space and if i scroll down i can see some other other sort of things and the idea here is i take a look at these incidents and maybe they've got resolution notes that are um that i can copy into the the actual case uh incident so i uh so i can go ahead and resolve it quickly so instead of doing that research it finds it so here's a very simple example of using the classification now how did we build that so if we go into all predictive intelligence classification solution definitions is where we define the model okay solutions are the trained models so i'll click there and then i'm going to do new and you can follow along if you'd like but my point here is the the actual and this is by the way folks there's mul there's a few other ways to create a classification solution this is our advanced interface okay there's easier ways to do this but this is our advanced interface and notice our advanced interface has four steps this is our advanced interface okay so i could say predict assignment group and i'm gonna call it for ai academy just so i know that this is different and then it's going to say step one please select a word corpus step two what are you interested in predicting what are your input fields and step four add some filter criteria okay so why don't we go ahead and uh and fill this out and then i'm going to talk about what's behind all of this okay because there might be some terms here like word corpus uh that you're not not familiar with and what is this doing or output field like what what do i need to worry about picking what's a good output field so uh in in classification and regression we can skip the step one word corpus okay and because we'll auto generate the word corpus and i'll come back to this and step two what are we interested in predicting so i'm going to go off the incident table i'm going to select incident and i want to predict the assignment group because that's what we've been talking about step three what input data is helpful to predict the output field so i'm going to click the slush bucket bucket button here and i'm going to grab short description for example but you could grab other other fields right what you don't want to do is like just mass select things without understanding whether or not it's going to help us predict the output field and that's what we're going to go into so go ahead and pick short description and then step four if you're following along pick a filter criteria i use for example create a date created by and then pick something that will get you to under 50 000 rows because if you want to follow along by the time you if you pick under 50 000 rows um like between 30 000 and 50 000 this should train in about 15 to 20 minutes so go ahead and uh do a filter criteria here um to get you under 50 000 rows and try to get between 30 and 50 000. we've got our processing language our stop words and training frequency and then go ahead and hit submit and train now i'm not going to do it because i've already i've already done mine but go ahead and hit submit and train on yours so that by the time we come back to this it should be trained okay so so now that we've got that model uh set up we're gonna dive into these different steps here all right so let me get back here all right all right so let's deep dive into the first step okay so we talked about um following along on your instance oh i'll come back to that all right so the step one so the step one we're talking about it asks please select a word corpus now i explained that in classification and in regression we will automatically uh generate the word corpus based on the input fields that you have selected now what is the word corpus used for well the word corpus allows us to select the vectorization method all right what the heck is that well the vectorization method converts all that text right you're picking short description or description it converts all that text into numeric vectors so that it can be used by our ar frameworks because our ai frameworks such as classification regression clustering all of them they don't take text they take numbers and so what that does is and here's an example of vectorization of hello using a our paragraph vector vectorization method it creates this big numeric vector and so it's not just converting the hello it's also when we when we run vectorization all these different data points here these different numbers show the relationship of hello to other words like goodbye right or greeting and so we do that because now it down to this it's a number we can plot those words into n-dimensional space and then we can run linear algebra functions again against it so things like distance so if i were to vectorize dog and cat i could run mathematical operations and see that these are very closely related and so this is where we are using why we're converting and why the word corpus what the word corpus is doing it's turning all our text into numeric vectors so that we can do this comparison so we can compare dog and cat and we know that pizza is not really is far away from dog and cats so it's not related now think about it in terms of you know your incidents coming in i've got an incident that says my email is is slow and then someone says my outlook email is not working so great we can convert all those words into these numeric vectors and we can plot them that they're on a in them in on in n-dimensional space and we can see that those two short descriptions are very similar to one another okay and so when you select that word corpus and again we just grab the default and we let it do it for us but when you get into the options which i'll i'll get to when we're done toward the end um you're going to see three different options you can select paragraph vector out in the field if you google it it's actually known as doc2vac and that's the default and then there's something called turn frequency inverse document frequency so this is another uh vectorization technique and this is we found this is very helpful if your text has a lot of repetitive data so think about like machine generated data like event data or templated data you're probably using shortcut templates to auto populate a short description so that's very useful for improving the performance of your classification model uh when you use term frequency inverse document frequency and there's something also called glove or global vector which is useful for training your models against industry-specific vocabulary so if you're in a manufacturing or healthcare organization um you you may want to try this out now you've got all these options i always recommend start with the default why you may ask well leonard why do i even care about all these different options and and i will be going through lots of different options the reason that you care is when you just use our defaults it may be good enough but i've worked with many many customers where you know it's they they want to improve the model and sometimes experimenting with these different options like these different vectorization options or some of the different advanced options will um improve the the performance of the model okay so i'm i'm to minimize screen bouncing i'm going to cover all this stuff and you guys can click and follow along so wherever i have follow we're going to come back to this once i get through some of these core slides and and and we'll we'll see it actually in the product the step two what are you interested in predicting okay so what is the output field in our case we were trying to we're trying to predict the assignment group but it's important before you you even go in and build that model if you want to predict the assignment group for example you need to understand and run a report and servicenow you need to understand the shape of the model of the field that you're trying to predict so here i've got the assignment group distribution a normal distribution works best and you want to avoid what's called a data imbalance like what we have in this first graph so imagine these bars are is a report where these are all assignment groups that handle password reset now if you look at this intuitively the model if you just feed it a short description as an input and someone creates a new incident with password reset where do you think it's going to go i'm asking that rhetorically well it's going to go to the winner because if we just use short description the password is going to fall it's probably going to the model is going to look and say well my only input is short description and i'm trying to predict the assignment group well this assignment group handles all the passwords that's the majority of them so i'm going to send it here now and in a lot in that that's typically not correct right because you may have working organization where you have a global service desk and you have like americas and asia and europe and they all handle password resets and if you only give it one input such a short description it's always going to go to the wrong place and so we need to understand that shape to say okay well we see that this group let's say it's america's handles most of the password resets we need to add additional input so we know if if someone is asked has a password issue reset issue from europe we want to send it to the europe uh assignment group that handles password resets another thing is so we want it to look more like the bottom graph here where it's a little bit more distributed evenly distributed another thing is the the output field that we're trying to predict um is we want to make sure that it doesn't have any null fields doesn't have any different languages in it you know if you're predicting for english it should all be english if you're predicting for you know dutch it should all be dutch and then shouldn't have any weird abnormal characters because that'll throw the algorithm off so that's the first sort of like advices and recommendation is just visualize your data visualize your your your the field that you're trying to predict and then what we're going to do is we're going to take a look at the input data that's the step 3. so as i showed before you need to pick some input fields that will help you predict the output fields and i showed an exaggerated example of where i selected a whole bunch and that's not good practice so you want to understand which inputs are highly correlated to the classification output and you can do this by running a script and when we're going to get to this i'm going to post this uh this into the chat so you all can try it you probably can't run anything right now because your model is probably training or or you can try this against an existing model but we'll we'll get back to this but basically what the script is going to do see here on the right i've got i'm trying to predict the assignment group it's going to say these input fields subcategory category etc etc it's going to give a model improvement number and that model improvement is the correlation between the input field to the output field and so the closer it is to one the more highly correlated is to predict the assignment group so we can kind of sort through some of the input fields and pick the ones that are going to help us um predict the the assignment group uh and and if you have any questions please do put them in the q a uh my peers on the phone will will answer it and i'll if we have time toward the end i'll i'll look through the questions that are unanswered and i'll address them um so we'll we'll get into this you'll you'll have a chance to try this out toward the end that script also does something where it takes a look at the input fields and it identifies the density and so when i said you know make sure your input fields don't have any nulls and so run that script to identify that so the closer to the it is to one means that it's it it's most likely 100 of the fields are populated right and so we can see you know short description description these all look good anything below that like alternative contact phone we may not want to use that because it means 40 of the data is blank okay and so i've got some other examples here of you know i'm trying to predict in english but i've got some uh chinese characters in there i might have some special characters in there so you want to avoid um that kind of data also if you are you want to separate out if you're running machine learning against event data machine generated data you want to separate that out okay so you want to have a model that will predict event data and maybe a model that will predict against human generated data and then the final step which is uh what historical data do you want to use so this is where you are when i talked about supervised machine learning you have to teach it right and so this is the data that you're going to send to the model to teach it this is what we call the training set and so here you want to make sure that you pick data that you know is relevant is current um that you that has all the answers that you want this model to learn from and so this is what we're actually doing and so what we do here is that in this final step right we went through four steps we're defining what we call a 70 to 30 split and and i i have some code if you had to write out the code i have an example of what we would do this this is python code one of the data science popular data science uh languages you can see here we're we're splitting out the the test side so we do this for you so in in our in servicenow we split this out but this is what's going on behind the scenes when you define that step four you're you're defining a training set in a test set and and because of that we know the answers um and you'll see that when we go and look at the the uh the model performance all right so we just went through some of the things behind the scenes and now what we're going to do is we can go into our instance and just explore some of this so um here was the uh here's where we started with uh predicting the assignment group right and you should have hit submit and train uh there's that word corpus and i'll we're going to get into this but if we leave it blank we're going to auto generate it uh if i go ahead and just uh save this we'll get the related list yeah so you have to have a minimum number of records i think it's at least 10 000. so let me try saving that again so i'm going to remove all my training conditions because i don't have a lot of data on this instance so i've got 50 000. and so when you do that you'll have the related links show up uh yours should be training but under advanced solution settings i can go into new and i can select those different algorithms like xgboost remember i talked about that earlier where if your data is skewed this is a great algorithm to help you run a more effective classification solution we've got the tf idf we've got some of those other solutions there so that looks all good now in terms of the word corpus i talked about this a little bit and the word corpus if we go all word corpus and we get into it you'll see all the default word corpuses and i'll select one of them but you can create a new one and this is where you can select the different vectorization techniques that i talked about again paragraph vectors are default tf idf is great if you are have a lot of repetitive data or if you have templated data that you're using to fill in the short description and then pre-trained is what our global vector or glove that's where you have the industry sort of term so this is where you would select the different vectorization techniques all right i'm going to pause before we jump in and take a look at the questions marcel's got that hey hey guys marcel and the bill uh any uh are there any questions that we need to answer or we are okay yeah leonard the the only question um comes from one of our audience members and the question was um where in the platform do we have deep learning and um you know others feel free to chime in but um you know our response is there's areas that kind of span you know the space of natural language processing deep learning machine learning etc and i you know in the q a i listed out some of the different areas where you may see some different models but i want to direct everyone's focus to not getting so much wrapped around the type of model but how to really engage with the model how to interpret the model and then how to understand the model's impact on the end user experience but i'll open it up to panel members to elaborate yeah i i 100 agree marcel i mean it's it's sort of like what we focused on earlier where we just really want to understand the outcomes that we're trying to achieve with the with the ai right we're trying to improve the work experience um and so um we really try not to focus on so much that we're using clustering and process mining we're really just trying to use it to identify patterns now um yeah marcel you're absolutely right in natural language processing we've got deep learning models um under safe harbor as we move into into utah we'll be using those deep models deep learning models also for classification as well and as well as in other areas yeah and leonard one one comment and i think this may be worth mentioning is because we're talking a lot about data and we're talking a lot about input fields and how they're going to be good indicators or predictors of output fields but do you want to share a little bit what happens if you don't have great data today meaning you're pulling data out of a legacy system or you're new to service now or you've changed your processes over the course of the last several years and maybe your data is in all sorts of different formats can you share any best practices on how to get started so in terms of so right you you want to pick the data that is going to help the model be the most successful now um you can experiment i think in our documentation we say a minimum of 30 000 records um to get started but uh you can bring that down but often what i i recommend is you may need a combination of you know the machine learning and maybe the traditional ways of predicting like the assignment group so for example if you only have 10 000 records but the majority of those can predict these five assignment groups uh very well but then there's maybe ten that it can't we could design the model to uh predict uh certain assignment groups and then default back to an assignment rule if the confidence is low so maybe we know most of our data is we've got 20 assignment groups and we've got 10 000 records and only let's just say only five of those we have the majority of data for well you can design it where you have logic where if the confidence of the of the model uh passes like maybe 70 go ahead and use the prediction for everything else go ahead and then use the traditional assignment rules so there's a multiple ways of tackling it that that one is um a common one where we have a hybrid approach where we'll use the machine learning model uh and we'll test the confidence of it if the confidence is high enough go ahead and use that prediction um when we don't have a lot of data and if it's not confident then we default back to either the human figures it out or we have some assignment rule logic that figures out the assignment group and then over time hopefully you build enough data where the machine learning can completely take over um that that analysis great point leonard yeah so it's important just to get started even with the records you do have and what we want to see is incremental improvement so don't feel like you have to get it perfect the first time out of the gate just incremental improvement is the the direction we want to see you trending so no more questions let her back over to you great thanks purcell all right so you'll get this uh you'll get this deck so you can click into it but let me uh let me post this into the chat hosted panelists so everyone so this is a this is the next step so all right cool all right so what are we gonna do now so let me i wanted to get through the slides just to explain some of the concepts now we're just gonna walk back to our model creation here so um so here we we had the solution definitions and we talked about you know creating that new that that uh the that new um solution classification solution to predict the assignment group we walk through you know the word corpus you can see here we since i didn't select one it auto-generated one and we also quickly went into and if you blinked you probably missed it we went into all word corpus and i showed you how you can define your own word corpus and select those different vectorization techniques uh that we talked about we talked about you know taking a look at the output field uh and seeing you know if it you know if the output field has any nulls in it that kind of thing if it is uh heavily skewed or imbalanced and then we talked about that winner take all then we talked about the input fields and instead of just randomly selecting a bunch of input fields now what we're going to do is we're going to go to this link and i'm going to show you um how to execute this these some of these scripts that will help you pick the correct input field so here at the bottom of this article uh you'll see you know this script and so you can run this now uh you you have to be an admin on your instance but you would run this script you'd copy this and you would go into background scripts leave this page and you're going to run it now don't don't paste it in yet it don't run it yet but i want to walk through this code here so what this is doing is it's looking at the incident table and it's got an encoded query a short description is not empty now your data if you're going against your instance you probably have millions of records so this you're going to want to change this encoded query um before you run this it's got a label and what this is going to do is it's going to find the eligible input fields for the assignment group okay and so i just wanted to show you where you would paste this um you probably don't have time to write the encoded query correctly and i put a pro tip in here to actually get the encoded query um you're going to find your solution definition where you uh you created you know you picked all those different filters and then you're going to go into the list view and you're just going to copy that query and you're going to post it into that code okay so um my recommendation don't do this right now because you probably want to you know write this encoded query correctly so you're not pulling like a million rows but what we will do is we'll uh go into this next piece so after that script is run we're going to run this piece and so when you guys run the script after you fix that encoded query you're going to go ahead and run this which allows us to retrieve the results okay and so up let me see here back on script miss something try here we go and so this is this is what i was showing you in the slide um so here we're predicting uh where we want to predict the assignment group in our classification solution and it's going through these input fields and it's picking out the ones that have the highest that have passed a certain correlation level and so these might be things that we you know uh think about maybe picking category right and it does take some intelligence right because you wouldn't just pick subcategory without category because you need the subcategory uh you need category before you can leverage a subcategory right as an input so we might leverage the category so it's going through and giving you some ideas of some other input fields that might help you predict that assignment group and down here is also showing you the potential like the input fields and their density okay and so it looks like all my input field all the all these different input fields are all good because they they have quite a bit of data okay all right so let's uh let's kind of get to the performance part and then we'll sort of wrap up okay so i talked a little bit about this i gave you some some examples and so this will be here for um for your reference the one thing i didn't cover was the confidence threshold so npi when you train the model and you run it there's a confidence level that confidence level has to be high at higher than the threshold in order for pa to actually write the value write the prediction into a designated field okay my whole point with this is i i write a lot of our algorithms in python um this is the beauty of it you don't have to get a data scientist to use the power of machine learning in the servicenow platform uh you can just out of those four simple steps you can go ahead and leverage that um that machine learning capability all right so we went ahead we trained those models but we're not done okay so we need to look at whether or not the model is effective so i'm looking here and i'm seeing that the estimated precision is 76 the coverage is 77 and recall is 56. is this model good enough to put into production and the way i always ask customers to look at this is look at 76 percent if we fed the model 100 tickets it was a 100 incidents it was able to correctly predict 76 of those cases and then i asked the question is that better than what you do today when you're doing assignment group routing do you route 76 of the of your incidents correctly and what percentage gets reassigned because if we use 76 that means 24 gets reassigned so in order to figure out if 76 is good enough you need to ask yourself well is it better than what we're doing today and if it's not then we would spend more time and we would invest the time to find additional inputs to look for better training data to improve these precision coverage and recall capabilities now i'll just use this extreme example where someone may want 100 precision but there's also this downside because precision will impact recall and so you may you may see this you will see this when you see the stats on the models what is recall right one concept i want you to understand right away is that precision and recall are inversely related so if you increase precision you're going to decrease the recall and to understand recall it's and find that right balance i have to this is the most technical slide in this presentation but you need two pieces of information you need to understand this confusion matrix where on the diagonal of true positive two negatives this is what the model is predicting correctly in the and the other diagonal false positive fault negative this is where the model is predicting incorrectly and if we look at this model and we say maybe this model is there to predict cancer so true positive means the model predicts cancer when the patient has cancer okay a false positive model predicts cancer but the patient doesn't really have cancer false negative is when the model predicts not cancer but the real value is cancer so in a false negative is where i want to draw your attention because of its relationship to recall you have to ask yourself when you do these models and you're looking at these statistics with recall a false negative the model predicts not cancer but the patient really has cancer is that that's is that good and i ask that rhetorically hopefully everyone's saying no that's not good so we want we want a high recall in those situations where we want to make sure we capture all possible um uh possibilities of cancer so a higher recall and and so the concept two is these are the formulas precision and recall if i look at recall all it is is the number of true positives in the numerator and the denominator plus the false negatives we can't change the true positives but we can change this false negative can shift and so if we just look at this intuitively and if we want a high recall and we want low because high recall means false negatives um means that we're reducing false negatives we we want like a higher recall because that means that we have um you know a low false negative count so we're we're not going to say someone doesn't have cancer when they really have cancer and so those are kind of situations where you have to ask yourself is recall important to you okay and so that's that's something that you just kind of have to absorb and you can kind of look back at that and back at the recording but just think about that cancer example when you're looking at your data uh you would relate it maybe to a security breach right in predicting security breach is it important to capture all potential security breaches if so you want a higher recall now here are the different again the different statistics around the the performance of the model it's really easy when you go in there to go and set the target value so here if i increase the precision it's actually going to go through and decrease the recall and that's why i wanted to make sure everyone was aware of that recall precision relationship and here we can actually go into the the train solution we can actually go into each of the classes so here i've got a particular class that i'm predicting and we can pick the precision and the coverage and the recall that i want so you can actually go through here and pick that and we'll we'll take that we'll look at that really quickly if we have time but in order to do that we'll go into um all predictive intelligence classification solutions let's just do that really quickly we go into all predictive intelligence and uh solutions which are the trained models here are those solution statistics okay here's where we can force the the different values so if i wanted like 100 precision i could apply that value but it's going to bring down these other values and then below here this is where i can actually drill into some of these predicted classes these are my assignment groups and then i can pick the precision the recall that i want and then i can apply that to the model the model will use what i picked there okay all right so finally you've got it all tuned you've got the model working the way you you want and you're ready to deploy it so how do you do that well you have three options you have um anywhere you call where anywhere you use javascript you can call the pi model and the benefit of this is that you've got full control of that model you can call all kinds of advanced capabilities um when you call a pi this way customers typically use it and they call it a business rule or they'll call it from a ui ui action or or a macro but you have full control here but it requires you to write some javascript code and use our ml api flow designer if you want to integrate this say like in the beginning i said integrate this into your workflow and you don't want to write any code there is a predictive intelligence spoke that allows you to pick the different models that you've created and then finally there's something called pi workbench which we didn't get into but pi workbench is a guided setup essentially that walks you through um a use case where maybe you want to predict the assignment group and it it goes through all that in a guided setup and then it lets you uh then you turn it over to allow us to do the integration of the model for you and so what's beautiful about this is it's really simple to integrate we handle it all but you have limited control now i won't get into these because we're almost out of time here but in the in the learn more section these additional classes go through that so here here's an example of a k21 uh knowledge class um that uses uh flow designer and uh to implement the solution now if you want to learn more about some of these advanced techniques you can also go into now learning and look at these predictive intelligence advanced topics and some of these other courses whenever you type in predictive intelligence you'll see a number of them they also cover some of those different implementation techniques so with that i'm going to shut up because i could talk about this for many many hours are there any follow-up final questions we want to ask before we wrap up marcel yeah leona there's one question um so the question is you mentioned creating separate models in the case of skewed data can you give a rule of thumb regarding the number of models we should have in production for example a few models is most likely um 20 models can make sense 50 models you're most likely overdoing it so any guidance there yeah yeah right you don't want if you're using assignment rules today you definitely don't want your predictive intelligence models to outnumber your assignment roles right so it um it um it's a it's some experimentation but i don't see like uh that that one skewed data balance example i gave with the password reset being handled by multiple global um geographies we were able to condense that down into like three models for predicting the assignment group but we started off with 10. but if you go through and you experiment and you really look at the model you should be able to bring it down so you're always wanting to bring the model the model count down because it makes life easier for you in terms of just maintenance and and avoiding that that um sort of that spaghetti code but but to talk about that you have a lot of resources marcel being one of them on this phone where if you reach out to your sales team and you have questions about the proper models the number of models and things like that they have access they can connect you to all the uh all the experts at servicenow that can that we're all here to just help you be successful with it with using predictive intelligence in your in your environment so um so i'll leave uh that's how i would answer that yeah thanks leonard yeah and just keep in mind simple is better um and leonard one thing to point out i know when you were showing your um solution statistics with your precision your recall and your coverage numbers um one thing that that we've kind of observed is how to take this to the next level so aside from the model the model tuning getting the model performant oftentimes we see customers struggle with how to communicate what kind of value is the model delivering back to the business and this is a great area to look at if you look at your coverage rate your coverage is really indicating how many of your records are being you know essentially solved for with this prediction algorithm so if you have almost 100 coverage think about your cost of ticket your cost of incident your cost of case um how much of that is being covered by the model and what's the improvement over the way you were doing it before so most likely a human intervention way the cost of that agent's time or the cost to the service desk so those are some rough estimates you can begin to apply to communicate to um you know supervisors business leaders on some of the value that the solution is bringing so just wanted to point that area out yeah great marcel so we appreciate i know where we were only supposed to talk for 45 minutes so and we're over by six so marcel you want to wrap us up if that's cool sure all right well thanks again leonard this was really really informative again keep visiting us every other week every other wednesday we will have a variety of topics for you and again head out to the ai and intelligence community forum for resources additional information we greatly appreciate your attendance and look forward to seeing you next time great thanks everybody