with another workflow Wednesday and today one of the things I want to talk about is what the heck is a major incident now if you're in I.T you can probably just check out because you've lived this you understand it completely but servicenow expands the Enterprise and so as many people across the Enterprise start looking for ways to improve the way of work improve improve efficiencies and look for ways to work together there's terms that come from different parts of the organization that we're going to hear major incident is one that you might have heard and maybe hear it from natural disasters or other things but within the world of work particularly within the world of I.T an MI a major incident is a topic that we'll often hear talked about a fair amount and you know that's what I wanted to talk about today and I wanted to use the fact that I have a personal story I get to share that with and explain it with plumbing and so let's walk into that first major incidents are when there is a service you're providing that goes down it's no longer accessible and and it's stopping work uh and it grows in scale as that impact to your business impact to your employees impact your customers grows and so these are often called Priority One it's stop everything let's figure out how to fix this so my Story begins of course in the middle of the night because major incidents never happen in the middle of the day well they do but you know the the stories that you hear and the experience are often in the off hours so for me and for my wife it started at about 4 15 4 20 in the morning we got a call from our daughter uh who lives maybe an hour away from us panicked she had to go and start work she had to be there to open up so she had to leave but as she's leaving she looks up to see her ceiling starting to collapse it's a major incident so she's calling she's like I don't want to do I'm freaking out there's something going on on the ceiling there's something going on in the house what do I do what do I do I've got to get to work and open this up and you know I'm on boarding into servicenow I've been here a little over two and a half months and in that time there's lots of training I had a mandatory training starting at 10. so this is you know there's some time there's you know less than six hours but there's some time and so we started thinking well what do we do do we call someone who do we call who do we get involved there's some really kind of stepping back as we wake up what can we do so the major incident is there's something going on very likely water because uh what causes a ceiling to collapse just like you would in the work world you know what's what's down this this application is down what could that possibly be which teams need to be involved and so that's kind of the first step is truly is something that impact is there's something wrong is there an incident and is it causing impact enough that it should be considered a priority One Stop everything else get up at 4 20 in the morning and drive and go and do this to be able to come back in time for a class it's the same type of planning when we think about major incidents what who what do how important is it for us to address this for in this case P1 the next step when you think about a major incident is trying to assemble the team and there's two two different teams that I think are important that we see in service now when you look at how major incidents are hap you know handled in the platform and enabled in the platform it's it's those that are are communicated to and need to be involved in the communication chain and those are involved in in looking at it resolving it restoring service restoring service is key one of the things they find and you'll hear it in this story too is the purpose of the major incident is to as quickly as possible find out how you restore service in some manner you might not fix the root cause you might find a workaround you might find some other way to do it but things will not be restored to the way they were before or the way that you'd really envisioned them to be done in the future same is true in this story and so I get up I get dressed uh the communication channel for us as my wife and my daughter and so you start figuring her out and here's here's what I'm going to do and and make a plan make a timeline of of when are we going to hear the next update who's going to be involved in resolving it who's going to be resolved involved in you know Communications and communication channels and what is the frequency of that communication with a major incident that's crucial particularly when you've got customers and many people involved we want a very accurate schedule of when there's going to be updates for this I need about an hour and a half hour and 20. I need coffee and and so I assembled my tool got dressed assembled my tools figure out what I'm going to need and got on my way and yes I did stop for coffee because that early in the morning I had to have some and so as as I went there I'm thinking through the layout of her house I'm thinking through what could cause this and that's just like your your implementation teams or your major incident teams response teams are going to be doing as soon as they hear about it is they're groggy they're getting involved and they're they're getting their caffeine and they're getting involved in the systems they're going to be thinking about all right what could take that service down where do we need to look is there any changes that have been done lately you know for my daughter this is a new fairly new house she moved in a couple weeks prior and so we don't know a lot of what we don't know and we had no home inspection done but that's a whole nother story that I don't want to get into but I'm not a fan of paying home inspectors they don't find stuff anyway with major incidents you want that team thinking about where should they be inspecting where should they be looking inside of service now if you have the configuration management database the database that stores all of the things that you care about to do your business to put it under change control so you can track it and monitor it that's important but what we've seen is users have understood that the relationships between all of these things matter that's CI configuration item to CI relationship this uses this which depends on this and that tree of relationship helps us understand what could impact things as that rolls up into what we call the common Services data model csdm we've got to love acronym cmdb csdm as it rolls up we can see there's different services how might that play out in my daughter's house it's really the services she has a functioning master bath in that master bath there's multiple things inside that region there's a ability to go to the bathroom there's a shower there's a sink and and all of those have water both water going in and water going out I'm electrical engineer by training and so one of the things I learned from my dad is the gazuntas have to add up to the gazadas yeah I know not so technical but uh it matters you know Plumbing just like electricity same type of concept applies in those Services those configuration items if you will in my daughter's bathroom have both Source lines going in valves fittings controls faucets and then they have fixtures that are collecting it you know the shower pan and the drain and the plumbing that goes underneath the shower the sink and the P-trap underneath it and the plumbing that goes from that as well as the toilet itself both the tank and the bowl and the drain that goes underneath it's embedded in the floor each of those in servicenow and our technical world would be configuration items with relationships to each other as we think about that team is the service goes down one of the most important steps to look at is how is your change management process being done do we have a track record of what was done when and if these this item here was updated what would be dependent on it your teams are going to be thinking about that what changes were done the day before was a team did a team just deploy a new set of firewall rules do they just you know replace a database replace a server was there some change made because in general problems don't show up without some change so what was the change for me as I'm driving down driving down the mountains it's going to live in Colorado and so driving down the curvy Mountain Roads enjoying that to kind of clear the mind but also letting me think through what do we know about the configuration items in the master bath that's right above where she described the ceiling as collapsing started thinking through where that could be you know looked like a recent Remodel and so it's possible that there were some Plumbing issues Behind the Walls that we should look at There's issues where possibly with the toilet or the sink maybe there's something there but I wasn't sure and so the first thing I did when I got there was started looking at what are the sources what's the impact why is the service down for me why is the ceiling coming down is it a flow of water is it just a collection of water and in here what I found it was a source of water was very very slow so I started shutting water sources off I didn't see anything moving I wanted to check that first shut the water sources off then I started looking just like your teams would do what else is there and along each of these steps we need to send communication updates to the team that we decided should be involved in Communications for me that's my wife and daughter right I made it to coffee I made it to the house I got in I found where the there's there's water and there's collapsing I I think I know where to look next and what I found I had to do is I had to cause some damage I had to get up on a stool because I was quicker than getting the ladder and cut back the ceiling paint that was you know hanging down like a balloon I had to cut that out then I had to cut out a section of sheetrock to see where was this water coming from in the ceiling as I cut it down I found lots of things that were wet but it couldn't tell exactly where it was it looked like it was more from underneath the toilet than the shower wore the sink and so I was able to send an update I think I found where it might might be coming from just like in your major incident as you start looking at the dependency tree you know we've we've identified well it's not the shower all the things that support their shower it's not the source it's not the drain it's it's not the sink that would be further over that would have come down somewhere else and it helps identify which teams and sometimes that's pulling in a different group hey it's not the the server team it's it's not the database team or it's a network team you can pull in different resources in my scenario it's still me but it's a different Focus and so what I found from cutting that sheetrock out is it looked like it was coming from the toilet couldn't tell if it was the source of the drain side I'm always suspicious of the source side because that's where there's pressure and so I went up and looked and what I found is I look very closely is there plastic fittings I am an old guy I prefer sweated copper and soldering those together because I think that's an art form and I enjoy doing it well I understand the value of packs and you know being able to do the plastic fittings in plastic valves just because of cost but as I looked up there one of the things I saw and one of the things that made me frustrated with our home inspector that we paid is we had a plastic fitting that came up from the through the floor with a shutoff valve and then on top of it is the attachment to the hose that goes up to the tank of the toilet it was very clear that that valve was at an angle plastic threads don't do well if they're mistreaded and so what happened was it looked like as we moved as we came in you know that that angle so we started using that toilet there was a very slow drip that mistretted valve was causing this slow drip and so over the two weeks of being in the house it's just drip a little bit at a time so did anything change the night before that caused her ceiling to collapse no it was probably a problem that had been started you know two weeks prior even before that there was just this very slow drip with water collecting knowing that and thankfully there's a Home Depot very nearby I measured the pieces understood what what I had to get to replace the valve and communicated that out with my team with my wife and my daughter I found the problem I think I have time I can go to the store and I know which pieces I need to get I'm gonna get it just like major incident inside of service now you'll have that Communication channel you'll meet with the team particularly as you start having a broader team you'll have status meetings to to check some organizations keep a bridge open constantly so you can communicate back and forth inside of servicenow you can chat as the on the Mi record so you can keep the team informed both as work notes internal as customer comments external facing to those that need to see in the communication chain um as updates are made you can shape your response in what's next for your response team and you can also shape expectations of when do we think we'll be able to restore service when do we think we'd be able to cause the stop of the problem because sometimes there's that sometimes there's you know not only a service is down but there's detrimental things happening in my case there was water leaking causing damage first was able to stop that yes in your major incident is there something going wrong is there data being corrupted is there something else being damaged can you stop that then after that it's how to restore service and for me it meant I stopped it I need to disassemble I needed a new hose because I didn't trust the threads within that and I also needed a new valve and I needed a fitting for the plastic plumbing coming out of the ground equipped with that I ran to Home Depot found the shark bike well use shark bites right any of those DIY plumbers um they work great if you get them on a nice clean cut pipe and so I was able to come back and of course you know I was able to get the new valve on put it in place reapplied water pressure with the valve closed and could see that there was no current leak so I wiped it all up made sure it was very dry so I could monitor it and then started to put the hose on that I got from Home Depot only to realize that I thought I got the right part but it was put in the wrong slot when we're under pressure often we'll try and pick things quickly one where this caught me I didn't double check I didn't I've known this before you have to verify everything you get from those large stores but Home Depot you know they had several I was looking for the right lengths and the right fittings and and the label was right and I grabbed one but it was the wrong size fitting on one end so I need to make another trip back I'm looking at my watch looking at my drive time up to my class I'm like I don't think I'm gonna make it had to jump on and tell my my teacher in my class again who's on the communication channel that needs to know I needed to update you know my manager and the the teacher in the class hey I think I'm going to be back but I may be a few minutes late working on an emergency went to Home Depot found returned the missing part found the right part came back put it in place connected it to the tank of the toilet and was able to apply water pressure to it verify that that there's no leaks then did you know a rough cleanup and was able to actually see my daughter she she came home and was able to talk through what I had found and talked through where we should watch we believe we've restored service but there's some things we need to monitor in a major incident you might find that same thing that as you restore service you find some way hey we've we've got this other server another copy of the database we can move this service here connect it like this use a different load balancer whatever it might be in technical terms your team is sharp and they'll find some way to restore service but as they do it there's going to be some things we need to watch This Server might not have as much memory might not have as much you know CPU power whatever it might be that we need to track usage and if usage starts to get too high we're going to have to step in and preempt it some other way in my case that was I think I've solved this problem here's how you shut it off here here's how you shut it off across the the whole house here's what we need to watch for so with that I was able to get on the road and communicate to my my wife and daughter yep I am leaving I believe the major incident is resolved we there's existing problems left right I cut a hole in the ceiling they're sheetrock missing what we did there is similar to what you do in service now there's some more work that needs to be done and servers now we'd create a problem record say there's some root Clause we need to do there's some repair that we need to do and so we'd create a problem record and identify here's what has to happen for us it says to note we need to find a handyman that can come out and you know put in a piece of sheetrock because I just don't have time to be able to do that to be able to repair the ceiling and restore that to final fit and finish with a major incident might be the same way that it's resolved but you need a problem record to track out how do you resolve it you know to restore service completely as it was before to understand where that problem came from your team as they resolved it might even not understand the root Clause there's more analysis that needs to be done to track that down and so in my case we got that all restored and got it all set the other thing we did is we we made sure we brought a fan down to circulate air to dry that out and knock on wood that has been solved but as you look at servicenow and you look at your environment one of the things that we see in service now is servicenow does change the world of work it provides workflows ways you can get teams to to collaborate together understand who's taking this next action and who needs to approve things who needs to be communicated with to be able to see what's happening to be aware of progress service now does that across the Enterprise it's not just for it anymore one of the products that I get to be involved in is the service called impact an impact for our customers that that use it assembles a team we call it the impact Squad the impact Squad is a team that's basically watching different things they're watching the instance his updates are being done they're they're checking to see how that's progressed much like in the building world you know it's kind of a building inspector you might have some inspections on code you might have someone come in and double check things you might have a home inspector that would actually be aware if there's a misthreaded valve I didn't I'll find a different home inspector likely me um but your team on the impact Squad would have that perspective that time to look and say here's some things we're doing on the instance here's some things that are happening in the ecosystem how's that going to impact your organization and they can preemptively jump in what we're hearing from our customers is instead of things becoming a major incident with a call at 4 15 in the morning the team is able to identify Things Early and stop them get ahead of it and say you know there could be some problems here we see a mistretted valve there's some problems here we see that you know there's coding that's not quite following best practices there's there's maybe some security issues in in the way this integration is being done there's some other you know interrelations maybe there's new capabilities on the platform that are going to impact some things you've done in the past that impact Squad can catch all of those things now many of those things right like I'm always leery anytime I say or hear the word all uh we'd catch many and one of the things we're finding is having that impact Squad having access to these what we call impact accelerators very concise engagements where we bring in our expert Services team to respond and look at things that could be a code analysis it could be an instance observation could be some other things looking at your cmdb or your how csdm is set up or security all these tools end up helping us have a much clearer understanding of your organization and how things are running in and the scenario that I described from Plumbing it would be I have good analysis I understand where there's water flow when there's not is there water flowing when I didn't expect it what would cause that do I have analysis of which branch of the plumbing tree if you will that that water is flowing when I didn't expect it to those are the types of things that we might be able to instrument inside of servicenow where you're watching different kpis and looking at server loads looking at database loads looking at other things and and having some early awareness that something's different and why that could come from your event event management system could from come your vulnerability systems there's so many different organ things within your organization that can feed us information and data about how work should flow and help us understand how to quickly preempt that with impact we help enable that and so again hope you enjoyed this workflow Wednesday that you have a better understanding of what a major incident is and I hope you you don't have a 4 AM call about Plumbing with a ceiling collapsing underneath you but just know that servicenow has processes in place to do some of that if you'd like to talk about that more uh hopefully I'm easy to find on LinkedIn if we're not connected do reach out but I am a Services account executive with servicenow I cover the Rockies yeah but I enjoy connecting with others if I can help you or I can connect you with someone else that can do reach out and I look forward to talking with you thanks so much bye