[Music] [Applause] so predictive ai ops observability into micro services and all that legacy stuff and i quote legacy because what a lot of the customers we talk to they're just getting the cloud they're lifting shifting into aws um in the way that we see the world is it's it's hybrid right and we're going to be able to to manage this very very forward-looking um you know microservices that you're putting into place as well as that traditional on-prem premise data center and begin to help you with that migration right understanding what those applications look like what is a good fit to re-platform into the cloud and move forward with so show me don't just tell me oh yeah make it relevant so everything that i'm going to show you um this is technology that exists within servicenow today except for one dashboard which is going to be displayed in grafana but all the data in that dashboard all those sre metrics are actually going to be pulled from servicenow and actually pulled from the servicenow aiops collector our agent client collector so you're going to see something very real so uh the flow of the demo just this is only two slides i just want to frame it up a little bit the idea here is like the way that we're changing the way that we're deploying software is changing right uh we are going to deploy a micro service that's actually part of a much bigger service and this happens to be our cart service v2 and what our cart service b2 does for our boutique application is it allows us to our customers to utilize stacked coupon codes right so we can save all the money so let me show you this thing really quick uh before we just a live demo right so here's my boutique application uh running in aws a little bit of a hipster thing going on they sell typewriters terrariums and record players barista stuff so this is a real live running website uh it's i'll show you the architecture it is a grouping of microservices that make up this online store application so so the first thing that we're going to do and i actually should have done it while i was out of powerpoint is we're going to deploy that change right and we're going to call this a devops change because this change is coming from your ci cd pipeline and it's going to have all that metadata along with it that comes with that ci cd pipeline who is the developer what are the what are they changing uh what other impacts right can we generate has there been security testing right so all of that can get packaged up and all that metadata about the change can get packaged up through the servicenow devops tooling and pushed into service now as a fully vetted change record and what we can do with that is we can make decisions right hey should we let this thing go to prod is it going to be a canary deployment or are we going to push all of this code to production right so now i'm going to simulate that uh everything else is real but the devops change and building all those tools a little bit complex so i'm going to simulate that with a script there is going to be a real change record there and we're going to simulate some other integrations with the devops tool set so this thing is going to go ahead and deploy i'm feeling pretty confident at this point so when we talk about a canary deployment right that's sending the canary into the coal mine uh let's shift 10 percent of our traffic over to this new stackable coupon uh you know part of our website so so our customers can ultimately save money so here's what my uh here's my cart service looks like right most of my traffic is going from version one i'm beginning to funnel some traffic over to version two um i don't know if you can hear my slack going off but but something's happening right uh something isn't going well i've introduced a configuration issue into my environment and friday as an engineer i'm saying oh my god what's going on i'm getting all these notifications my website is throwing up 500 errors uh is it the dependency and the tokenization for my api gateway like what do i do right so we have all this stuff it's happening it's not good we start receiving all these metrics right and our metrics our logs our events um and then we start talking about observability right what is the latency what is the traffic doing are there 500 errors uh is there saturation in my environment what i can tell you and all these slack messages are telling me something's drifting right i'm getting all these alert messages on my phone in slack microsoft teams whatever you're using wherever you want to be but something is drifting from normal so i have a problem so let's talk quickly about observability and what it means to service now and you mentioned oi uh earlier i think we have a pretty good solution for oi now um i think one of the limitations with oi was that we were relying on third party metrics right and it was rather hard to integrate with well now we have our own metric collector and our own event collector called the agent client collector which in this environment is actually deployed as a sidecar application as part of a kubernetes cluster and that's receiving all the metrics about this boutique web application we're also taking the logs in right so all the health logs and logs are where we can begin to work in the yellow right we know what a normal log pattern for our application looks like we also know what a abnormal log pattern begins to look like from our application so based off of this log data that we're also streaming into servicenow we can begin to detect anomalies not only from the event in metric data but the log data as well so what comes out of the events and the metrics we get anomalies alerts and we begin to see patterns and what do we do with those anomaly alerts and patterns we cluster things together right so we take um alerts from your apm tool from your other event management tools from the data that we're bringing in from our metric collection our log data and we cluster those using you know time text and that service aware topology that servicenow is so good at and we create one meaningful actionable incident now that said there's a whole bunch of other workflows that go on with that hey i just took my customer facing application down you know what i need to report out to status page that this thing is down so right that's an api call and a workflow i need to let my sales organization know that there's an issue right so i'm gonna i'm gonna page them i'm gonna hit them on slack just a workflow in part of this overarching outage so then what do i do i collaborate we swarm on the issue we try to automate the resolution and we try to gain insights right so we take it full circle so that when we try to deploy cart service version three um we don't take down the boutique website for the hipsters so now with that i'm gonna get into the demonstration itself and show you a couple of experiences one from that sre persona one from the persona inside of servicenow and i'm gonna show you how we can use our automation engine um and you know servicenow to get some situational awareness and actually roll back uh this issue i'll show you my boutique website right so we're serving up 500 errors something is very wrong let's take a look at uh you know what i'll show you my slack messages first right so here's all those messages on slack i start seeing a swarm of alerts coming in now i'm an sre at this point so i can do a few things here right i can acknowledge the alert probably best practice right hey i'm taking ownership of this um do i need to go into service now do i want to look at the application service map absolutely but what i want to do is actually i built some instrumentation based off of the data served up in servicenow around this uh application um and i'm going to pull up my observability dashboard and these are all the key performance metrics that the servicenow aiops agent is bringing into the servicenow platform but instead of being displayed in service now um you know but the sre team kind of built this dashboarding around this application um inside of grafana so the first thing to notice here is if we look at the last i've been talking for a while so let's look at the last 15 minutes we now had an extreme uptick in 500 errors right we went from almost nothing to almost one per second we've also had an extreme downtick in the request volume for uh you know our online boutique application so here's a bunch of other metrics around um you know the application itself as well as the micro services deployed as part of it and if we talk about you know that micro the micro service topology this is what it looks like displayed in grafana but this this graph or this uh this service map right here of that micro service this thing was completely discovered by servicenow's uh discovery engine right so we've mapped the service mesh topology and we're also taking that alert data i'm going to show you what this looks like in servicenow as well and we're you know binding those alerts over the issues with the various micro services that make up boutique i'm also giving my engineers and my sres the visibility into what's going on in servicenow so i have log analytics alerts i have secondary alerts because i'm deduplicating and correlating across monitoring systems and then i still have i have this other you know just breadth of graphs and knowledge about every all the instrumentation around my application and then here's some very specific in a chart the metric data that i believe are the kpis that make up my application this is all being pulled from the servicenow agent client collector um whether it be from you know prometheus inside the kubernetes cluster or the agent client collector actually looking at the application itself so let's go into service now and see if we can fix this thing so this is our service view and i think pablo showed you can i kind of like future state well this is what it looks like today right so we can see all of our services across geos he mentioned it earlier right we can slice and dice this data any way you want maybe an sre team wants to have a very granular view of even just the micro services that make up my boutique application but maybe we have an overarching more holistic knock view too of our entire operational estate so let's look into the boutique service and kind of take a look at that same service map but look at it inside of servicenow so we use servicenow discovery to actually map this this application which is a collection of microservices we're also feeding in the alert data from you know the traditional monitoring systems or the apms and that metric uh event in log data from the agent client collector and why these are lit up red is because we're binding those various alerts and metrics over the services that are impacted uh within our environment so if i take a look at cart service i can see all the information about this kubernetes uh service and if i look at this i can actually see any of the alerts that are coming in from my various systems of telemetry about the various microservices or the boutique web application so we'll dig in a little bit further into this grouped alert so here are you know four alerts that we've grouped together right because we want to dedupe and correlate across monitoring systems whether it be hours or um you know any third party apm that you're bringing to the table the data dogs of the world what not um and if we take a look here we can actually get a little bit deeper and see where they came from and what they are so we do have an app dynamics alert and the rest of this came from you know boutique our product catalog service and cart service and this tells me one thing what i did as a developer deploying that update to my micro service i've impacted somebody else's service right my product catalog service has also been impacted and without this level of you know tying the business back to the service it's really hard to do this in other tools the next and the probably most important thing is i have my probable root cause because we're one platform and we're using that devops change and we're taking in all of that data throughout the cd cicd process and we're pushing it into service now we now have all of the information about this change right so this all came in from a machine rather than a developer needing to push it into service now it's just you know different um you know it's just different uh data points from various tools all along your ci cd pipeline but we have information on how to potentially roll back this this change as well so if i go back into my um my alert record i'll talk through a few more things right here's all the configuration items that we've impacted um the alert executions right and this is the most important part and i'm going to show you this a little bit better in a second this is service now and it's power to have digital workflow so this one's relatively simple right we have this group of alerts let's go ahead and create an incident in this case it's probably a major incident because we need to alert out to the business there's probably some sort of sla attached to it there's probably going to be some sort of post so all of all that change and all the things that are happening right now can be tracked within that incident for when we do need to run that post-mortem and then the next logical thing to do here right as that that developer who made the change is likely going to be to run the remediation and roll back that deployment are you well are you rolling back the deployment are you just pulling the ten percent you sent to the canary deployment yep so i'm gonna i'm actually yeah i'm gonna roll back the canary v2 micro service um and i'm gonna change the configuration to direct all the traffic back to the v1 service um the point here though is you can go ahead and this can be this can be your own workflow right yeah um you need to leverage like best practice here right is probably to create an emergency change and maybe not let the developer who deployed the bad code into production auto approve and push more uh more code into production that could also potentially cause another impact so we probably want to change record in here we'd probably want to manage your approval before we actually go ahead and do that roll back but then we can orchestrate with any number of tools right whether it's your ci cd pipelines uh ado jira jenkins etc um all the work that needs to go into actually back rolling back this change it can be well orchestrated within a servicenow workflow using out of the box integrations um and if we're talking about maybe like a more like a more traditional legacy problem in your infrastructure you know think about um you know infrastructure changes as well maybe we need to call an ansible job to expand disk space or expand a lawn or clean out temp files all of that can be done and i think you guys said it earlier we see you as an automator of automation tools it's what servicenow and their their workflows is really good at so hopefully at this point we'll go ahead and refresh the flow uh we've rollbacked our change um hopefully boutique is running again it appears to be all right we can buy typewriters and terrariums if we go back to our observability dashboard we'll notice uh let's even get a little smaller here we'll notice in the last minute or so our 500 errors have been reduced our request volume has went up um and all of our services are healthy again now all of this data right keep in mind is all coming from servicenow all coming from our metric collection capability and if we go back to that operator workspace things are clear if we look at that service map one more time the alert should be cleared and we've remediated this issue [Music] [Applause] [Music] [Applause] [Music] you