logo

NJP

Off-loading System Event Processing to a Dedicated Queue

Import · Oct 12, 2023 · video

[Music] hey everyone it's Justin thank you for watching and welcome to my house in this video I'm talking to you again about system events particular setting up a new queue for events that are taking too long in the default queue that's what this video is about this is more of a kind of advanced move so if you're not familiar with events processing go check out my previous video there's actually a service now now learning lab and course around creating events in service now I found that to be really helpful so you might want to go check that out before you go and do this but I want to complete um kind of my investigation and findings I got these two event processing alerts and one of my demo environments and when I went in it looked like I have the long running event which is taking more than 1,000 m milliseconds and in the previous video I showed how important that was so I want to basically offload these and somewhere else so I'm going to do that here with you we're going to go find this event and we're going to go set the queue for it so PA job DC ended okay so let's go into system policy and we'll see the event registry the event registry is where all these events are registered and if I go into let's look at that name again pa. job. DC pa. job. DC um there I have the ended okay if we open up that event registry we'll see that the Q name for this event is blank that means that this event is going to go into the default queue and when it goes into the default queue it's going to take based on the history here longer than 1,000 milliseconds actually look behind my head there you can see the recent event stats all longer than 1,000 milliseconds so it looks like from what I can find I'd love you to leave a comment down below tell me if you know any different that when there are frequent high volume events or long running events they should be in their own queue in the previous video I showed all the different cues that had longer running events so we're going to do the same thing with this one I'm going to create a new que called PA Q performance analytics q and uh so we'll save that and now if this event gets fired it should go into that particular queue um now the of course the stats aren't going to show now but when the event gets fired it should go into that queue now it's not going to actually do anything that queue is not going to get processed unless I set up a scheduled job to process that new queue it's just going to pile up in that que and not get processed because nothing's filtering or listening to that Cube so we got to complete that step so the way I did this in the lab and the way that I'm going to show you here is we're going to go to an existing scheduled job we'll just type in scheduled job in our menu there and what we're looking for in scheduled jobs is the system scheduler scheduled jobs menu item right here scheduled jobs so we're going to go there and the documentation uh in the um documentation here use custom cues to process events and the lab that I did say to use an existing scheduled job and there's one for text indexing so I'm going to search for Star text and you can see text index events process now this was one of the things where it gets into one of those advanced moves if you do this the first thing you want to do when you do it is insert and stay so you're not modifying this particular scheduled job so I'm just going to call this um paq uh process and I'm going to insert and stay really quick so now I'm off of that other um job that was running now point out that this job is running every 30 seconds as I copied it let's go look at the other jobs and see how often they're running so if I go in here and search for uh I'm going to change the filter and actually let me open that up again this text index events and I'm going to look at this script Glide event manager that's what I want so when we look at this we're going to filter for the job context contains uh I already forgot is it Glide event contains Glide event let's see make sure that works yeah so these are all the ones that are processing those cues and look at the Times there behind my head we got 30 seconds 5 minutes 1 second 15 seconds 5 minutes and 30 seconds so 30 seconds seems to be the most common wow some of these are narrowed down to even 2 seconds well that makesense the sidebar uh notification process but it looks like we have some variability around that so let's go to mine the paq process that is performance analytics and in particular it's running about data collection jobs and this is a demo environment we want our demo environment to show demo data but is it important to run every 30 seconds I don't think so so I'm actually going to change this from 30 seconds to every like 2 hours right I don't really care how long or that this runs every 30s seconds I mean we're demoing maybe a couple times a day but not nearly that much in fact I could probably change this to every 12 hours and uh and get away with that and that's way it's happening in fact let me do that I'm going to set this to every 12 hours the next thing I want to do to make sure this event gets processed is we're going to put in there instead of text index CU we're going to look for the PA q that I just created so I'm going to change this to paor qprocess and we'll save our work we've got a new name the next action is going to run at 8:30 Pacific Time or 8:31 Pacific time I think because it was running every 30 seconds but now it's going to run every 12 hours so in a minute here when or in 30 seconds this should update to 8 + 12 would be 20 2000 hours um or 2031 hours it should update to so let's go ahead and reload the form and see if that got our new interval of every 12 hours just going to refresh this a couple of times because I'm talking there it goes it just updated to the next action it's going to run is at 2031 and it should pick up anything that was in that queue from that system uh from that event that we saw the event alert from the the new queue I created called paq so let me show that real quick just to be thorough here the event registration and right here for pa. job. DC we're putting it in the paor Q so that's what we want so the last step that we have in here is to go to our system events and open up that alert that is triggering so we can see I'll narrow I'll zoom in here that this has happened almost 70 times on both of them since this new instance has been running 67 on one 70 on the other so I want to reset this alert um so if we go ahead and open up this first one here we're going to click this new reset alert button so it should notify me if this alert gets generated again I'm going to ahead and reset the alert there we go and I'm going to close that window now and let's just refresh this screen and now I only have one left it's the same one PA job DC ended okay so I'm going to open that alert and we're going to reset alert and now I'm going to get notified if this happens again in this little clip what I'm showing is the queue actually queuing so later that evening I went and recorded the queue showing everything was sitting there ready before it got processed we'll show that here in a second all right we're here about 24 hours later you can see I have no event processing alerts on my system events and jobs dashboard which was I first identified that issue so that's good we're still at zero and if I look at the Q um that I made paor Q you can see there's a bunch of stuff that's ready that happened overnight um but if we scroll down here you'll see that this stuff from yesterday after I made that change all got processed starting at about I think it was um 8:00 p.m. or no what 11:00 p.m. on the it's going to say 10:30 here which is plus 3 hours which is 11 or 23 13:30 I don't know I'm confused on the time but it got processed the interesting thing that I'll point out is when this was running in the default queue my processing durations were really high um and so after that first one got processed here on the bottom um at 5,688 milliseconds the rest of them seem pretty reasonable um we'll see how it goes well actually there's one that's an outlier there uh the last one at 4,329 but the rest of them were pretty reasonable so um I'm glad it's over in its own que now and now it's only processing every 12 hours which feels more appropriate for the type of environment that I'm supporting um so everything's working successfully I have no more events no more alerts and uh I think I have everything I need so um yeah I guess I should end the video um I hope you found this video helpful if you did please like Please Subscribe or share with somebody who you think might be interested in troubleshooting events or long running events in their service now instance and until next time don't forget to always be [Music] learning [Music]

View original source

https://www.youtube.com/watch?v=BloH4JnuMEo