into session seven I'm will Hallam I'm one of your hosts so just uh we have a really informal agenda on these sessions if you haven't been to one before I'm doing my best to make sure everybody's unmuted or not unmuted but has the ability to unmute so feel free to either use the Q a or uh the chat or just take yourself off mute at any time to ask questions like to keep things highly conversational so uh we'll just kind of go over uh the purpose of these sessions some host introductions then we'll uh get into our Tech Deep dive topic and close out with uh just uh open q a why are we here so the the reason we started this series is just to try and bring together people from servicenow people from servicenow customers who are somewhere along the journey to Cloud native I.T practices to open an interchange of ideas challenges questions answers with a particular Focus over how the servicenow platform can assist in that Journey we are so um not joining us today because he's at a customer engagement is my my counterpart Mike Gallagher um just a little bit about Mike he's uh 25 plus years in technology he likes to say if it works with ones and zeros he's probably touched it at some point in his career he really loves working on cloud native Technologies especially kubernetes and he is a fairly fanatical practitioner of Brazilian Jiu Jitsu my name is Will Hallam I'm an advisory solution architect uh focusing on our itom product Suite I've worked in it for over two decades currently focusing on it operations management all kinds of automation type activities with a particular Focus around Cloud public Cloud kubernetes anything that lives in the cloud is something that I've generally had a little bit to do with and in my spare time I like to hang out with my family and play hockey and video games so now we'll get into the tech deep dive topic for today which is uh per our uh per our title um lightstep and now platform better together so uh before I just launch into that and hand things over to my co my guest presenter today I'm just going to put up a quick poll just to try and get a little bit of a temperature check as far as um everyone's familiarity with open telemetry and kind of where uh folks on the call are with regard to observability in general so uh per per the the title of our series beers with Engineers I'm also going to take the opportunity to open my beverage for the afternoon which is a Lagunitas hazy Wonder okay so we'll let those answers kind of filter in for another couple minutes and in the meantime uh Adam or are you out there I am perfect uh do you want me to just keep clicking through the slides or do you want to click through it yourself now feel free to just keep clicking through and I'll I'll have the fun narration of saying next next that works for me um okay so I'll just kind of let the poll keep going in the background and hand it over to Adam to start with a brief introduction to light step yeah my name is Adam Johnson I'm a Senior Solutions consultant on the lightstep side I've been with lights of about two and a half years so a year and a half prior to the acquisition of servicenow and previously was on the implementation and post sales teams but have recently joined the pre-sales uh side of the house um like I think we have a slide on the a little bit of the history of open Telemetry and lightstep on the the next slide so I can talk um so light step uh was founded out of Google and Twitter um our Founders came from monitoring and creating those systems um probably most famously are one of our Founders VHS was a co-author on the Dapper pipe Dapper papers that outlined what distributed tracings uh goal was Google's approach to it um and really introduced distributed tracing into the mainstream of observability signals um we survived as a venture-backed company for a period of time and most relevantly were introduced to the servicenow portfolio in 2021 but lightstep as a standalone SAS is ingesting over three petabytes of daily uh data and really a thought leader in the distributed tracing space and to touch on open Telemetry and kind of just parallel open source project and land um instrumentation has always been kind of a separate challenge within the observability space because observability is really about being able to explore your data and get insight from it and tell you where the pain or stress in your system is and the root cause and how to fix it is kind of the goal so in order to do that effectively you need to have data to look through and uh the previous slide mentioned that we co-founded open uh tracing um we can see here in 2016 uh lightstep is one of the founding parties of open tracing which is one of the first open source apis for uh implementing open tracing and as we follow this timeline uh down uh most notably the 2018 and 2020 points um Google released a instrumentation uh project for metrics specifically with open census and open Telemetry is the combination of both of those projects so the next major release of open tracing and open census really is what open telemetry was so that's kind of the timeline of the open source project lightstep has a proprietary SAS solution that does analysis on top of that but this is kind of the instrumentation timeline okay and to speak a little bit more about what open Telemetry is um the goal of the project is to provide high quality ubiquitous and portable Telemetry to enable effective observability so specifically open Telemetry is focused on Telemetry and the idea here is that you can have Telemetry from whatever system you are running using consuming and the Telemetry is just present within the data and done well and we see this can you actually hit the next slide I think we have logos popping up on the bottom um okay do one more here yeah I'm stealing from some uh a co-workers presentation of mine uh that had a Super Mario 3 uh theme to that uh if you click one more time we'll see the logos pop up but in order to um promote this vision a lot of companies are actually on board and taking this up so we can see with uh platform level tooling like your Cloud vendors or Splunk or dynatrace or the creators of Frameworks and libraries that they're actually including open Telemetry instrumentation as part of the platform or framework so that you as a user all that you need to do is say turn on open Telemetry and send your data somewhere and this is really the the core value that open Telemetry is is really trying to um accomplish and cook so being able to just have the Telemetry baked into your whatever tools you're using is the real magic here and it does have really great uh pickup uh um in terms of take up and adoption so these these companies are making very significant contributions into this and this this vision is becoming reality um go ahead and click next but the last thing to say as the slide Fades away is within the cmcf of which um of which show the plum trees apart it's the second fastest growing and largest project behind only kubernetes so there is incredible traction Community effort attached to this project uh and to kind of review and pick up the pieces uh can you actually click twice I think there's two animations on this um once more the idea just to get this kind of all at once open tracing started as the combination uh sorry go back one open Telemetry started as the combination of open tracing and open census uh targeting distributed tracing and metrics as the Telemetry types that's been expanded to incorporate um many different formats uh Prometheus and most recently elk made a elastic made a major contribution to the log specification uh dictating what log formats and parsers are going to be available within open Telemetry so again we can really make true this vision of being able to capture uh three of the the primary signals that observability tends to rely on in metrics logs and tracing um so this is this is a very big effort across many different open source projects and companies um and why this is such a big deal so a little that's a little bit more about Open plenty um next slide now please unless there are any questions are there any questions on this framework stuff I'm happy to be interrupted okay continue to raise your hand or just interrupt if you need to um but to talk a little bit more about these signals and kind of the history of the usage of them I think is really critical because we as as lightstep spend a lot of time talking about distributed tracing but we do incorporate we have a metrics platform and we can incorporate logs into this single set of tooling um just talking about the components of this I think really helps frame how to approach observability problems um and looking at these different signals and what they're good at um I think really helps people get an understanding of how to tackle these problems so um can you click I believe twice there's two animations here and I'm happy to talk to them at the same time a log is a very basic uh and foundational concept within Computing it's it's an instruction that the computer did at a specific point in time I'd expect most users with any introduction to technology to be familiar with logs and writing something to standard out when your application does something but this is probably the most primitive and straightforward means of debugging software the power of these is one people generally are familiar with these so it's straightforward to be able to interpret and use these uh the second benefit is that they are extremely explicit with regards to what happened um you can get a very clear picture by following the logs to see exactly what commands are executed and why things happened the real challenge is in a cloud native world with thousands and thousands of machines running isolated workloads that might might relationally be connected but from the Computing perspective or just isolated components that are executing it is a mess to be able to string these all together not only from a pure volume standpoint if you talk to anyone that pays a log vendor Based on data ingest will tell you this is very expensive to do at scale but also from a findability perspective trying to say my application is down or feeling stressed and only being able to take this broad stroke of searching through logs is really cumbersome and we often observe incidents using only logs to take on the order of hours to days whereas introducing other Telemetry signals can often reduce that down to um by 80 90 percent is not uncommon to find uh next slide please so uh historically the next thing that was introduced were metrics um these are fairly widespread at this point and really we've seen the advantage Technologies like nagios and things like that where you you set up simple monitors and the idea behind this is you are looking at a measurement over time so the example on the slides uh that I I haven't spoken to enough is about a morning routine um and in the previous example the logs were just talking through every step of our morning routine just everything you do write it down what metrics are attempting to do is in that same context okay take just the important notable events and either count or measure how many times they happen so in in the morning event how many times did you make coffee how long did it take you to make coffee um those kinds of things in a more Computing based example CPU or memory usage or network ingests are more actionable and common metrics to have but the idea here is that they're fairly High Velocity you can take these every few seconds and they really describe a specific system very well you can alert on these if CPU or memory is being consumed at a dangerous rate you can do the use metrics to forecast when am I going to fill up a volume and give yourself a proactive alert so you don't find out when the volume Falls over and these are excellent at being lead indicators for a horizontal slice of infrastructure the main challenge in distributed computing land in which we live uh in modern applications is that these are just a horizontal slice of data so they are typically only describing a single system um so you would measure CPU for a kubernetes pod or maybe average CPU across some node something like that however that doesn't describe or give any information to Downstream or Upstream workloads so if your application is suffering you will see that you are suffering but you would not be able to tell if it's a dependency that introduced a new version that is throwing all these errors you have to have a human created relational view of how that works and it becomes really difficult to paste to piece that all together the other challenge that metrics typically have is with cardinality um the the way that these are typically indexed is a combination of their their name and the tag values and it becomes very very costly to add High cardinality tags to this and what I mean specifically by high cardinality is very unique values so something like a kubernetes pod name inherently it's ephemeral it's coming up and going away and you are going to have millions of these over the course of a month or at least thousands of them is a very common occurrence however trying to auto complete from thousands of IDs is both slow expensive and very hard to find the right one so there's another cost to mention where this becomes impractical to try to imprint your application topology into and that's just historically been a pretty bad practice within this just from an operational cost Ambit and a true dollar cost because vendors again tend to price in that dimension um I think you'll need to click twice because I do believe there's another animation on the screen um but I would like the next slide one more so incomes distributed tracing and the role that distributed tracing plays in all of this is tracing is effectively a structured log so if you're looking at this and saying hey it looks a lot like a log it is but there's two real magic components to this in that there is a trace ID and a span ID oh there's also a parent's plan ID but that's kind of how you build this relational model where you communicate a uh yeah you can continue clicking through it I think the attributes are going to be the last tag that will load so stop when you get there um here's where you communicate uh the the thing you did the name or the workload um the span ID is the instance of this happening so this is the almost 43 000 at the time you made coffee um and it's part of a morning routine for uh January 1st of 2022 if I'm interpreting that time stamp it starts and ends and there's a relationship built into this so the real magic of this is that you can start seeing the actual execution environment in the view of a request um so someone comes to an endpoint and says give me this resource we will see every single different service or microservice that goes into compiling this and environment variables for each of those runs so this is a very powerful way to actually understand are there upstream or Downstream applications of Downstream or Upstream applications that are introducing problems or is the problem actually with the system that is alerting so understanding scope and shape of these is really what distributed tracing excels at answering and the other interesting thing is distributed tracing systems do not really suffer the same way uh and from cardinality that metric systems do so we tend to encourage users to include High cardinality metrics like a customer ID or instance so that you can ask the system show me every time a or like show me performance by customer is there any specific customer that's being impacted in addition to show me infrastructure slices that are running hot or taking a long time so that's um and the the downside of traces is is very similar to a log we often talk about the um a sampling is a core part of distributed tracing if you profile every single request it can be expensive so we can make decisions on to how to choose less data and there is a little bit more data to parse within a trace so if you want very real time lead indicators for alerting metrics tend to be the best use case for that so that you can catch things before they before they fall or you can Auto Scale based on more infrastructure aligned components whereas tracing focuses more on an application perspective um I see three questions in the chat so I'll I'll start addressing these uh Anthony is asking can lights up work in non-case deployments like an Azure spring boot app or container service or even a Lambda um yes absolutely uh distributed tracing assumes a uh doesn't assume anything about your application um these are typically included by just including a library alongside this and what happens is that Library will say for every unit of work create a span and then send it off to a back end so it can be generated from lambdas monoliths um really any code base in which you can include a library can be represented as doing this so I know there's Amazon being on board with open Telemetry has specific uh tooling for including instrumentation for Lambda code and makes it really easy to actually send to a back end they of course offer x-ray but um they are able to just export traces from Lander runs and Rahul I see Andrew Andrew's a colleague of mine on the light step side he's been a light step I think four years um so he's in process of answering but to give you a verbal uh Trace ID and span ID or typically uh generated by those libraries that you would add and just to touch on instrumentation because I know how do we collect this data is always another component um it's a very analogous to installing an agent um on your code for Java it's a post-compile thing you just run a jar alongside your jar um for python or JavaScript you just say like include open Telemetry profiler effectively and it will run alongside your code you can manually dress up traces metrics whatever but the automatic collection is actually very good and very easy to use so I I'd really Advocate you start there foreign next slide please I think this is going to take us to okay uh just to Showcase a little bit because I I assume distributed tracing is probably the most novel concept for most of these used for uh General audiences um overall and translating what a trace looks like into the interpretation of like the the visualization that you get after this I think is really powerful so um this slide is is about terminology but what I want to focus on is a trace is the overall request so a trace and a request um functionally mean the same thing and a trace is made up of spans or units of work or components within this so a client or an auth component or a billing component might make up these I might have their own individual spans and all of these view together are a Trace uh can you click to the next slide how these end up being viewed is typically in this like inverted pyramid uh shape yeah I've heard these as flame charts or um inverted pyramid charts but the idea here is that a request goes to the client um so there's a client transaction that starts this is a user hitting an endpoint or making a request of your system and it will Cascade through all of the downstream dependencies uh the things that we get inherently from this is what's what systems are involved how long did each call to these individual components take and typically on a live tracing system if you clicked on one of these individual um uh spans it would show you the actual attributes environment variables and details associated with that exact execution of of the request so this is how they typically are viewed um I believe there might be one more slide if you can click just to Showcase what this looks like in lightstep this is this is a screenshot from the tool and we can see here the notable things to point out is on the on the top where we have that inverted pyramid like shape we can see a black line running through all of these different lines what that is is where CPU seconds are actually being spent so we do this critical path analysis to tell you where the execution is spending and taking time if we're in the product we could actually click on one of these to investigate that like for instance that big long green line at the bottom if I was interested in performance looks like where we spend most of our time but just this example alone we can see on the right hand side all of the attributes and even the log events associated with this specific run so we can see what type of client work that the initial front-end web request came in we can see the HTTP status code that we returned we can see the log events associated with this and what the payload actually was alongside this so someone can reliably recreate this and replay this style of request and also we know that all of we have that context when trying to understand what the performance looks like um yeah I I was just highlighting the attributes and log events um this I think will just point to those specifically I think there's two animations here at one point attributes in one point into logs uh the thing that I did not mention is attributes are searchable and aggregatable aggregable um whichever the word is but you can you can aggregate your data and ask questions based on this so very common questions are of show me a breakdown by HTTP method to this endpoint and then we'll show you the get put post um breakdown and the traffic and performance of those or show me everything that came in on Android and is this slower than iOS or something for example those types of questions tracing very natively answers and I'm sure you can picture a solution based on a bunch of log messages but it requires complex writing and programming to do this kind of thing in the actual firefighting of an incident being able to see what happened on a long-running request or why an error came up or even where the error originated is immensely powerful and cuts down that MTT whatever measure instrument that customers are taking normally mean time to resolution or detection and helps them address those problems much faster Carlos asks is there a benefit to complement tracing and metrics with a logging solution or is that challenge that normalizing traces metrics and logs is hard to cross-reference or correlate um absolutely almost everyone I know runs a logging solution alongside tracing and metrics tool we tend to talk about these three pillars though I will acknowledge people to use other signals besides these three but almost everyone uses all of them where you use one you typically use the others as well because they are answering distinct problems I don't I don't have a slide for this but I will draw your attention to at the very top of that circled Red Box on the slide there is a workflow links component what that allows us to do is actually parameterize URL requests to another tool a very very common one is a log search so what we can do is said say during the time span of this Trace do a log search for either this trade specifically or other things that match these attribute sets so the interplay between these tools is really powerful and what I actually normally talk to is people tend to respond to events with metrics metrics again are very good at describing uh single horizontal system and distributed systems problems normally aren't just one machine they are several machines or several Services interacting together so once you look at your metric dashboard if you don't know what's going on or we're starting or scaling that individual service it doesn't solve the problem take a look at tracing because you can start understanding what the scope of the problem is where the source of the errors are coming from the magnitude of slowness or error rate that you're seeing and very quickly classify the event correctly or not and then the handoff then is typically going from traces to the logs of a specific service like tracing will answer what service is actually going on what's the real problem and what's the real impact this is having and then once you know there's a specific service with a specific error that's being thrown in logs logs are that detailed view that you need to understand exactly what went wrong I think this is the end of my slides and I think that's everything to cover so we it is the end of your slides um any other questions about what Adam's covered so far okay I'm not seeing any hands Rays anything outstanding in the chat or the QA so we'll keep moving forward um Adam thanks for that great overview hopefully you'll uh you'll stick around and keep me honest as I talk about the Better Together story and uh and show a little bit of an example of how we can complement lightstep with the now platform specifically in a a self-healing type of scenario and I did notice there was a question there was an anonymous question in the Q a about um kind of a road map question about when we might see lifestep integration with HLA health log analytics which is part of the servicenow itom Suite um and so if it weren't an anonymous question I could certainly kind of set a follow-up to get some more detailed information from our product team but given its Anonymous um I would say you know there's definitely options to do that today um open Telemetry does include a login component and there are ways to take that same logging content that's going into the lightstep platform and send it into HLA at the same time and then with some of the stuff I'm going to talk about as far as existing and future Integrations between lightstep and the now platform one of those one of the capabilities that that enables is visibility into what lightstep sees as well as what's in your cmdb all obtainable Within servicenow and so with that dependency mapping established log entries that health log analytics picks up and flags as anomalous would then be able to be mapped back to the corresponding light step service so in I would say we've got some capability there today and it's only going to increase and improve as we go forward which is kind of a nice segue into this slide which is just highlighting the ways in which combining light step and the now platform provide you know better than something better than some of the parts so the first point I want to touch on is just the fact that lightstep is the best of breed observability platform it's got some major differentiators compared with competitors in in that Arena it it has unparalleled Telemetry ingestion capacity the you know atom you touch on that in your slides how many petabytes of Trace data the platform is adjusting every day and that's something that really nobody else in the industry can stand up against today that everybody else has to drop data in order to just stay in near real-time currency with what they're feeding in widestep also offers the ability to pull in the correct data for the correct use case as Adam touched on logs to support root cause analysis traditional metrics which are kind of just table Stakes at this point in this um in this particular area and traces which support that high cardinality huge set size cardinality is basically just a fancy way of saying the size of a set or the size of an array and the solution to that where metrics tend to fall down is spans and distributed tracing and that's light steps bread and butter the second point is we have existing Integrations with our item Suite there is an event management connector which will take a web hook notification from lightstep when an alert threshold is exceeded or when it goes back you know when it clears there's also a service graph connector which combines an event management piece with cmdb integration that's currently available on the servicenow store in The Innovation lab going forward because we're all part of the servicenow family those synergies are only going to increase and become more sophisticated and evolve over time and what you're going to see is superior integration between the two products versus Solutions which rely on third-party connectors or Integrations and the third point is the the concept of one hand to shake by combining light step in the now platform customers have one hand to shake in their AI Ops and observability tool set along with a Consolidated roadmap for future function and feature and this one kind of resonates with me particularly because I've been dealing with uh some challenges with um with a service graph connector that connects to a third party observability platform and it turns out the API on that tool is not really very friendly it's very chatty and you have to make literally thousands if not millions of calls into the API just to figure out which data you need to pull down and the result of that is uh challenges with large-scale implementations because the API just can't provide the data into servicenow quickly enough to provide a fast turnaround time on import jobs and so that with that kind of Real World Experience not too far in my rear view mirror the idea of having a Consolidated tool set that all fall under one brand tends to be attractive to me this slide will just kind of illustrate where that Better Together story plays in in the context of an overall AI Ops implementation and we'll start we'll go left to right we'll start with the business services that are captured in the configuration management database varying criticality levels they all have components that are essential to making them work so the first step of getting your hands around that is pulling is is essentially populating a cmdb with the metadata around all of those parts whether it's Cloud objects applications running on in Cloud on servers Etc and what REI op solution can do is it can pull logs events and metrics natively from from those those sources and that's kind of our uh our first step in drawing from the wealth of information that's out there but can be difficult to sort through especially when there's some kind of a P1 outage going on top of that we layer observability and monitoring tools such as lightstep our own agent client collector monitoring solution as well as many third-party tools and so what that produces is is this stream of individual data points and the challenge with that is each of those individual data points is left to their own devices would result in a separate actionable item whether it's a task or an incident some type of a blob that someone has to look at figure out what to do with it and then fulfill it resolve it close it out what our AI op solution does is it shortcuts that part of the process by taking those individual pieces of data correlating them together collecting them into groups based on their relationship to each other as a result oftentimes serious issues can be predicted and prevented before they turn into the serious issue that's brewing and that's just related to the fact that things like health log analytics can kind of detect the ripples in the water minutes hours before they escalate into a customer impacting event an AI op solution like this helps identify root causes it can prevent 35 percent of P1 and P2 incidents and accelerate mean time to resolution as much as 40 percent and this last piece of the animation is a good segue into my demonstration which is going to show how you can use our flow engine to automate the remediation the Fulfillment of these actionable items okay so now what I'll demonstrate is an example of how lightstep can feed into the now platform to initiate a self-healing type of activity so what's up on my screen right now is one of the panels of the lightstep user interface and light steps basic taxonomy consists of projects and services so a project is a container which can contain one or more services this is a an example project that we use for demonstrations like this one which has a number of services which you can see on this left hand side panel which are instrumented using open telemetry and open till the open Telemetry data is being sent into lightstep which is then providing a number of ways to look at that data depending on whether you're in a monitoring type scenario uh you know forensics root cause analysis Etc so I mentioned we've got two different options right now two different Integrations with lightstep the one I'm going to cover is really going to focus on our service graph connector because that's kind of a One-Stop shop the there's a native capability via a the lightstep alerting functionality to send a event notification into event management that's kind of encapsulated in the service graph connector which also provides a way to populate your cmdb with data coming out of light steps so you see all of these Individual Services that are visible within lightstep what that service graph connector does is it instantiates those services within your cmdb and that's kind of what I'm going to show you here so the one of the advantages of the service graph connector is it includes uh kind of internally documented or self-documented guided setup similar to some other capabilities within the platform and so um the one I'm going to kind of highlight is the fact that it gives you a built-in instead of having to follow kind of some documented steps which which are fairly straightforward this adds an easy button to creating that event management notification web hook by virtue of it it's just built into the guided setup this launches a flow which connects to light step using credentials that you populate uh up here and establishes the web hook and what that looks like on the uh servicenow side is events coming in that look something like this and so as you can see I've got a lot of events coming in from certain lightstep service graph connector and the specific events that come in are defined via defining an alert and so the way an alert is defined is actually initially by creating a stream so in order to Define my alert I created a stream just saying show me information about the update catalog API call that's coming in from the iOS service and you can see we've got this is a you know this is a demo environment so it's got code that causes some aberrations to occur on a regular Cadence so you can see the latency and in turn the operations per second increasing on about an hourly Cadence and then I just established a threshold via this alert definition and so to make sure I get plenty of juicy events coming in I just set a threshold of five operations for a second in a five minute evaluation window will generate an alert and then going back to my events you can see the alert triggered events coming in coincident with the main kind of prescribed demo data spikes so the next step towards accomplishing a a self-healing capability is to create the action for servicenow to take when it sees this type of alert so the way I set that up in my example is I did a couple things one of the things I did was because this uh this lightstep data set is test test data there's no actual kubernetes cluster running that I can directly interact with with my my demonstration platform so I just created my own and I created kind of dummy services that align to the services that are included in this demo data so what I did was I just created a git repo and I defined each of the services as a kubernetes deployment just to kind of simulate what would be on the back end of a real application and so if I look at uh look at my demo project within the scope of this my test kubernetes cluster I can see I've got a whole bunch of PODS running one or more for each um for each of these services and so in order to simulate the action of taking steps when a open Telemetry threshold is breached I created this brief python script which will scale my cluster it'll scale a given deployment based on some simple command line parameters uh just a quick side note I did write all of this up in a community article and I'll send the link to that out along with the slides on our wrap-up email that will send out a few days after after the session that'll also include the recording link for uh future subsequent viewing so in order to kind of connect the now platform with this script that I've created I did a couple things I maintain a copy of this git repo on my mid server so that the git the mid server has access to the both the deployment files as well as the the scaling script and then I created a custom action Within flow designer called scale Kate's resource and it takes a few parameters so that it can operate against the appropriate kubernetes resource and then it just passes those parameters through to the copy of my scaling script that lives on the mid server once I'd created that action I created a couple simple sub flows that the event management subsystem can call when it detects an alert matching this particular scenario and it's as simple as I just put a little log step in there just to log what was going on and then I make a call to my custom action passing it the appropriate the resource name which is going to be the service name that's alerting within lightstep I just put in a an increment uh this was a quick example there's all kinds of capabilities to have that you know scaling increment be determined heuristically for example based on other prevailing conditions time of day um the magnitude of the of how far out of spec the alert value is ETC and then it just passes the up Direction since this is the scale up subflow and scale down subflow is basically the same except it says I'm going to scale down and then one uh in the today I learned Department as I was working through this example I discovered that our alert management rules will only trigger when something goes into a not okay state so in order to trigger a step that would scale things back down to normal I just leveraged a flow which triggers off of an alert transitioning to a closed state and also you know come that the alert is coming from the light steps uh service graph and then it's the uh the appropriate metric so after I put that flow designer content together the way to tie it into a event management is to create an alert management rule and so I did that over here so just in case anybody's not familiar with alert management rules they're pretty straightforward to set up and the alert filter you establish a criteria under which this rule will apply so in this case I said if it's coming from light steps service graph connector and it's this operations per second metric and the state is open or reopen then I'm going to go to the actions tab and launch this scale up resource and then like I mentioned uh because alert management rules will only fire when something goes from an okay state to a not okay State that's where this uh this parent flow that I created to trigger when an alert gets closed out to initiate the scale down activity foreign and that was it that was all I needed to do and now to kind of see it in action I'll just go to My Graph here so this is uh repeating demo data so the last Spike occurred um earlier in the hour so what I can show you is I've got uh actually have a little Loop running on my cluster so this Loop is just running some get pod activities on the iOS service that's looking for pods that are servicing that iOS service and so you can see [Music] here at quarter past the hour we went from a single pod to four pots and if we go back to the light step graph you see the time stamp for the spike in latency and Ops per second occurs at about a quarter after the hour and then that persists until about 20 past and then going back to the my pod census I guess maybe it's closer to half past because you can see it transitions from uh at around 25 past it transitions from before you can see those four are terminating and it's back down to one and so just real quick because it's it's pretty cool here's a example of the uh the kind of dependency enrichment that we get from the uh cmdb piece of the service graph connector so this is and I recognizing it's a an eye chart um but this just depicts what the CMD what knowledge your cmdb will have about your services which start from the light step okay here's the project and then here's all of the services that are running within that project and then in turn [Music] it will show you which kubernetes in you know in the case of kubernetes which kubernetes resources are tying to that light step service so in you know an incident type scenario or change management scenario to truly see that impact of something that's visible in light step or saw or a change that's going to take place what could that impact in your you know your devops teams and your SRE teams this is a really cool capability