and get our slides going all right welcome everyone thank you for joining us for session 11 of cloud native beers with engineers I am one of your hosts will Hallam and with me as always is Mike Gallagher uh just a quick rundown of our very informal agenda uh we'll just cover why we're here what the idea is behind this webinar series uh do some brief intros then we'll go into a tech Deep dive talking about health log analytics and stream followed by uh some live demonstrations and then we'll wrap up with open q a why are we here um this somewhat existential question uh the the answer is um the idea is to start to form a community of servicenow customers who are at a similar place in their journey to Cloud native and to share experiences and solutions that we found leveraging the servicenow platform to facilitate that journey and to make managing your Cloud native estate easier so now some host introductions Mike so Mike Gallagher um been around the the technology world for a long time um lately right I've been focused heavily on this intersection of servicenow and Cloud native capabilities um super excited about what we're doing these days and uh today I am going to be drinking a pumpkin spice latte right from uh spice trade Brewing because you know tis the season for everything pumpkin but it's a pumpkin spice latte beer 6.7 not too shabby was thank you sir uh will Hallam uh I'm an item architect over here at servicenow been in technology for a pretty decent amount of time um focusing these days on it operations management automation uh all the different Cloud native and cloud provider related things I really enjoy automating repetitive tasks if I have to do something more than five times or so I'm generally trying to find a way to automate it and in my spare time I enjoy hanging out with my family playing some pickup hockey and video games um oh and I am going to be drinking a juicy ale from Saranac to 4.8 which makes them fairly crushable for uh one of my favorite weekend beers and um yeah so cheers so so with us today we have we have uh a couple special guests from uh company called cribble so um cam Alex if you want to just kind of give a little brief intro for your own selves before we uh get started with the main program sure uh came here I've worked for Frugal I'm the director of technical alliances and um here to kind of spread the goodness of what people can do to help out for snag customers that are trying to get data into HLA and I am I've been with Brooklyn now for two years uh and I'm currently sitting freezing my butt off apparently in a wind tunnel here at Star Hill Brewery in uh I'm playing Virginia drinking their little red star little red blue star coffee cream nice nice hey folks uh I'm Alex Kane I'm uh alliances engineer under cam uh I do technical Integrations and uh have just live in code uh I came from background of operations work and development work spent several years at Splunk and workday uh now just a little bit over a year now at cribble and uh I am sitting in my house in Raleigh drinking a uh 1985 juicy IPA awesome love ipas okay without further Ado um what we're going to start with is is uh just a little bit of a setting a baseline for the two components that are the focus of our session uh Mike's going to start by covering some of the basics of health log Analytics yep and and before I dive into this too deeply just really quickly I I made it so that you guys can come off mute and ask questions if you'd like um feel free to do that as you guys know we've always liked this one this is an interactive session so don't hesitate to jump in if you have thoughts so um health log analytics and the shift in predictive AI Ops right really it in on the left hand side is kind of the older model where we've built out these metrics and then we figure out here's what the threshold is and then when that crosses and some things breaks we spend a bunch of time troubleshooting the logs and we go back into the cycle of okay we're going to build a new a new threshold in order to track this problem only that problem probably never happens again right um and one of the things that we talk about is as you know services are getting more complex that every event is a Black Swan event every event is a one in a million and how could that have possibly happened and yet you know we have black spawn Black Swan events happening almost every day so you can't stay out in front of it in the traditional monitoring workflow which is really where the predictive AI Ops with AI and ml comes into play right on the right hand side now what we're doing is as we're ingesting these logs we're understanding what's normal what's appropriate very very quickly we then can predict issues from those log based on patterns and the whole goal right ultimately is to to really have a reduced me time to repair but also to have a lot less toil around like how do we actually solve the problem and manage a bunch of stuff that isn't really necessary or providing value to fixing things so we move on to the next slide here we'll kind of walk through a little bit more about the capabilities itself right so um as we start to look at this right we're ingesting all of these events and all of this data set from all across all of these various different sources right and so um health log analytics is just one component of the predictive AI Ops platform right so it's another data source that we can use in order to correlate these things together and ultimately the goal is to then once we have found hey we have an issue if something has awry you know has arisen now we can actually Implement automated remediation against that incident and be able to solve the problem as as quickly as possible and while you know reducing human intervention and and the the swivel chair efforts right I always like about I'm going to talk about the human correlation engine right the goal of this is to take the humans out of that correlation engine do a lot of that automatically and then be able to repair it as quickly as possible so we can eliminate outages all these outcomes are here on the right hand side right that's the whole goal so when we go on to the next slide we'll we can dive in a little bit deeper so how does the health log analytics portion of that functionality work right this is an example of us ingesting log data from a particular log Source right so we're seeing hey this client has prematurely closed the connection right those are normal patterns that are occurring on a regular basis and then this this error pattern is another pattern that we're finding to be a new thing so if you click and finish building that out will right the what it does is it understands the text in order to build up hey here's what's going on here's here's what's normal right and then this spike in that particular log pattern there is hey this is this is an alert um that has occurred because this is an anomalous pattern that's outside of what we expect it to be and so therefore we can generally alert on that before um a normal monitoring issue can catch it right so instead of constantly chasing after hey this thing happened last time which caused a break and now we need to build a rule in order to try and catch that break before it happens again right now the machine is learning what's normal and identifying patterns that are occurring outside of normal and potentially creating incidents around that that way we can catch it as quickly as possible so let's talk a little bit about how we get the data into the platform and what it looks like from an architecture perspective right so on the left hand side here are the kind of the customer premise area on the right hand side is the servicenow data center instance capabilities right on the left hand side we have various different log shippers and various different log repositories that all need to feed the data into the mid servers in order to be able to get that data into the platform and it's very similar to how we do with event management or other capabilities generally we have a dedicated mid-server that is ingesting that data and that data comes in from a very specific stream and we're looking for specific components or specific data that then gets ingested into the load balancer that sits in front of the servicenow platform and there's really the key thing to understand is there this that now Glide um component in the top left corner there of servicenow that is your normal servicenow instance right and that handles all the UI and all the functionality for your normal servicenow work the other three components right occultist metric base and this doc store DB which is really elasticsearch um those three are actually totally separate servers within the data center that are just sort of attached to the servicenow instance and that is then um the importance of that is that that is actually um splitting that off and then in ingesting that data into those separate instances in order to prevent any impact to the primary servicenow platform right because the the AI engine and the time series data and that like longer term storage capability um hey well we're seeing that um yeah my bad no worries um so um uh all of that data is all ingested into those separate platforms and it won't impact the the actual usage of the main servicenow platform which is a really important functionality because that's all fairly compute intensive um uh use cases right so it's built and designed to scale separately and individually from the servicenow platform and also cause um as as little impact as possible on the actual servicenow platform usage cool so really really high level overview we're ingesting log data from the mid servers on the left from all the various sources and into the load balancer and what's interesting about all the sources there is that all of those sources can be routed through and Views crippled in order to do sort of that centralized aggregation and and routing standpoint was and and we'll have them explain that since they'll do a much better job than I will HLA before we move on sorry Health lag health log Analytics okay over to the team then all right awesome thank you Mike uh yeah so uh go and give you a quick technical overview try to overdo this slide um go ahead and think on the next slide for me this is literally what does uh we collect data from any source and we write we can collect it and transform it and also reduce it while it optimized and then send it to any destination in this case we're talking about sending it into a detailey or how far that works the benefit of having a triple do all this to it for you is that you have one place that you can manage the data collection of data the selection of data as well as routed to long-term storage so the good news here is that you can take data throw it into a data Lake whether that be an object store S3s mostly and later on down the road if you need to replay it because you've run out of attention space on your blogging platforms you can do that so this is what our affecting what our apologies architecture looks like go ahead and click on the next slide so it's a little deeper into kind of what we do so we have a concept of a leader node and worker nodes leader node is effectively where all the configuration changes have for people log in to make their changes those changes actually are stored in a git repo which are then pushed down to our workers our worker nodes can either be bare metal they can be VMS they can be containers this whole thing can be containerized and we can actually take this as a way to um collect this data through the worker Mode work engineer and then ship it to whatever destinations you need to do I'll see if there's a question so good question about the trade-offs what we do is we are all about getting the events that really matter into the log analytics platform of choice so there's a lot of times where there's the moral values there's data that is um there's unnecessary data that at the end of the day if you send it into your laundry platforms it can potentially create additional loads so so by eliminating that you have fast restrictions and the other piece of that is you can also route a full Fidelity copy of the quality events into an S3 bucket or on drugstore and she can then later replay if you for example Maybe uh took out a field that was was really necessary for your analytics so you can replay it back in and still have the forward notice so good question yeah I ran into that actually just as I was putting together this demo um I started out kind of drinking from the fire hose and just sending a lot of data in a lot of slightly different formats into HLA and once something hits HLA there's a little bit of adjustment generally that's required to kind of give HLA the right hints so it's pulling out you know that the correct time stamp the correct severity Etc and so it was very useful and and we'll get into that later when I kind of walk through the the demo it was very useful to be able to use cribble as kind of that that front end filter so I could focus on all the logs from one of my micro services for example and then get that all nice and tight and and working well and then I could go back into and kind of add additional services in a very deliberate fashion as opposed to um having to kind of sort through it once it hits HLA because that can get challenging the AI will the the AI does what the AI thinks it should do and if you give it a lot of different formats of data all at the same time it's you know some sometimes it's it's a bit of work to kind of pluck the right pieces out to get you know that first micro service effectively um effectively monitored so good that that's just a real life example of uh being able to kind of get that inspection on the front end before it hits HLA when you send the data into privilege you actually see what the database so you can actually see a sample of the data as it's coming in so you know what you're going to be taking out what it looks like before and so after the actually process the data so hopefully you can see it before you click go and start setting it in about fields quick question yeah so these are the common use cases that you can see uh honestly for people that are just getting data in routing they do this by almost like low level easy things to do with programming but typically is a very hard to do in real life so time stamps making sure that the events are properly formatted um those are the kind of things that you start to see um we do a very good job of giving you that power and ultimately if you have data going into a system you can optimize it so that you're dropping the field that you don't need formatting it in the proper format that it needs to explain routing it to multiple destinations if you have a security team that needs to them and you need operations team music from HLA you can route the same data to both destinations uh so there's a lot of really interesting things that we can do enriching data adding asset information adding goib information uh while helping reduce the noise give get rid of things like personal website identified information API in logs so you don't log that information other systems and then finally again archive and replay date all this with one product that makes it very easy to kind of manage your your data any questions looks like we're good all right I think this is the last name that is basically to help anybody that is doing povs targeting like sales Engineers that you know one of their problems is getting data into their platform we can obviously help with that and just make that whole process of showing the value of the actual like HLA um faster than having to work about the plumbing the data into the system okay and I never knew that this is actually an animation so good to know yeah I realized that about two minutes ago so finally there are three ways you can get uh you can either use our criminal file which is free uh you can ingest one terabyte of data per day it's available in US West USPS one um it is free forever just log in and start signing data in we have a downloadable product we can download and run it locally again a fair amount of data per day and finally we have uh this also available on Marketplace so on AWS Marketplace go download it deploy it in your AWS account and start playing with it this is kind of just to show what we can do we're exposed uh we're taking in sources we process the data through pipelines and then based on certain filters route the data to the important to the appropriate destination through the appropriate plugins any questions and then finally here is your lead behind I would strongly encourage anyone that wants to play through motivator or sandbox it's the full-blown version of our product along with an explanation that have music so you can actually go Leverage The Sandbox environment join our community feel free to talk to using this slack all of our training is free it will always be free so you can go get yourself uh certified in criminal through our University and he's also create content packs to help the community be out there's a new data source if you want to develop you can create the content backboard and then finally curious is a great place to go see what other people are doing and answer questions or ask questions in a form with like-minded individuals all right that's it for me all right thanks cam El all right uh so now we're going to the demonstration portion of the program so what I'm going to cover is just a couple example scenarios where I found that cribble plus HLA makes my life easier can make our customers life easier so to do that I'm going to introduce you to the basic elements that I put together for this little demo environment and then once I kind of show you the the pieces I'll show you how they how they work together and uh cam Alex as I get into kind of the cribble stuff please uh don't be shy about um augmenting what I've done with the actual kind of you know cribble official best way to to do things full disclosure I'm kind of just uh I've just been figuring out cribble by feel it's it's been fairly easy um the the doc I found the documentation to be really helpful and um yeah it's just made kind of Plumbing these things together a lot easier than it would have been otherwise so just running down the basic elements of my environment uh since we're all about Cloud native here at called native beers with Engineers I started with a kubernetes cluster so I've got this kubernetes cluster here which I then kind of threw a couple example workloads at the the most interesting one that's on here is uh based on the Google Boutique git repo and it's running a bunch of pods which represent a bunch of microservices which all function together to stand up a little demonstration Boutique for ordering various uh bespoke objects so this is running this cluster is running in eks in in AWS and because AWS kind of um has their own idea about how to run a kubernetes control plane I couldn't just directly access the control plane logs using generic kubernetes approaches they kind of force you to go through Cloud watch to get your control plane log information so in order to ingest logs from my control plane I set up a Lambda function which would ingest those logs when they hit cloudwatch and I just added a Lambda subscription to the corresponding cloudwatch log group for the cluster and so this is just uh this is well highlighting is not going to work because that color is just terrible so this is just a small python script that I wrote which essentially is taking Cloud watch logs uh decoding them decompressing them and then forwarding them along to cribble and the cribble destination is just populated via a URL on this particular line of code here um my plan is to put all the little code Snippets and and uh bits of collateral that went into this into uh Community articles so don't feel like you've got to like screenshot stuff or whatever if you want this stuff to refer back to later I'm gonna put it all online and we'll include the link to that when we send our traditional wrap up email for the session so that takes care of my control plane logs in order to send the logs from the individual microservice pods I use the tool called fluentd they are kind enough to provide several um manifests which you can which you can use one of the great things about cribble is that it speaks a lot of different languages and one of the end points that cribble automatically comes with when you use their free cloud um uh you know cloud-based tenant is an elastic search based endpoint which will just sit there and listen for elasticsearch connections and as it happens fluent D comes with a manifest which will deploy fluent D to your cluster with an elasticsearch endpoint and so all I did was I went into this manifest file added my herbal host as an elasticsearch host and added some authentication down a little lower just so random people couldn't connect to my uh kerbil tenant and start sending logs into the elastic endpoint and fired this into my cluster and so now I've got a fluent D pod running in each of my nodes and it's just happily pulling all of these basically doing the equivalent of a coupe cuddle logs on every single pod just kind of automatically essentially it's tailing the um the pertinent output file that exists on each of the the cluster nodes and it's just sending that along to Criminal so that takes care of getting my logs from my cluster into cribble and so now I'll go over to our cribble tenant as I said we are using the cloud-based offering I had done some work in the past with the AWS Marketplace offering that was pretty easy to set up as easy as that was this was even easier you just go you self-register you get a tenant it automatically comes with a bunch of standard log ingestion endpoints already set up and listening that you can start sending stuff to so I didn't even have to set up the elastic endpoint it it just came with it so the way cribble breaks things down is data comes in on sources it's routed through various pipelines and then it gets sent to destinations and so my sources that I'm currently using are the elasticsearch API which is pulling in data from the all of my pods that aren't part of the control the control plane and as cam mentioned one of the nice things is you can see live data as it's coming in so this is live data coming in from my cluster and I can see how kind of breaks it down by default and then from there I can manipulate it as needed in order to get it to look the way I want it to look by the time it hits HLA and I found this was a little bit quicker than just using the native HLA methodology which once I get into the HLA you'll see is it's a native kind of JavaScript interface it doesn't have the same interactive um point and click type user experience and I found that really helped me to streamline getting a bunch of data going into HLA so the other sources that I had to configure were um what's called a raw TCP or sorry a raw HTTP source which is ingesting the data from my Lambda function which is passing that's kind of my little Gateway that I had to create in order for data to come out of cloudwatch and get into cribble so actually that's probably a good time for me to start injecting a little chaos into my cluster so Mike had pointed out this this pretty neat um cluster workload you can run called Cube Invaders and it basically creates a little Space Invaders interface but as the little spaceships you know goes back and forth and shoots at things these are actually pods in my cluster that it's killing as it hits them with the you know with the projectiles that it's firing and then the little computer icons those represent my two nodes in my cluster and what happens when one of those gets hit is the cube Invaders pod will launch a chaos pod in the in the node itself which kind of reeks a little bit of Havoc with the uh with the node so now I've got kind of some Mischief going on in there it makes it more likely that I might see some control plane type uh traffic coming through on this raw HTTP endpoint the control plane is normally if you're in a steady state it's not necessarily terribly chatty and so here's an example of all the data that's coming in through that control plane log channel and then the last source that I've set up is uh what's called an S3 collector and for per the name it allows you to point fribble at an S3 bucket which contains log data which has been stored from any source and tell it hey go out and troll through that S3 bucket and find anything that contains log data and bring it back into [Music] the stream product and then it handles it just like any other stream of log data and so I right now this is pointed at a bucket which has a recording of events coming from my cluster from a previous date and again there's a lot of visibility here I can run uh the collector in what's called preview mode which is pretty self-descriptive it it pulls the data down and shows you hey here's an example of the data that would come out of this bucket if you were to run the collection routine and so again it breaks down the contents of each log entry in a nice navigational format Json is your friend with all of this stuff um if it's already in Json then this is the kind of experience you get where it kind of compresses your keys down and then you can expand your keys to see What Lies Beneath them if but but stream is completely format agnostic whatever format this stuff comes in on can handle it and transform it into whatever end State format whatever end State set of fields you're really interested in okay so now we've got our data um [Music] what do we want to do with the data before we send it to where it ultimately needs to go and that's controlled by what's called packs and Pipelines so packs are pre-built processing modules that cribble provides they are also sourced from the community and you can also write your own and what they do is they just provide a basically a drag and drop capability to say this is the kind of data I'm pulling in and you've got predefined transformation rules which will parse that data split it up if it's a particular format that sends multiple log records multiple log events in as part of uh a Consolidated payload and a pipeline is essentially a chain of functions that accomplishes the same thing it's just the pipeline is something that you build the pipeline is just more granular uh a pack is basically a set of functions that are bundled together which you can invoke a pipeline is in practice more more um bespoke for your particular situation so an example of a pipeline that I created which was fairly simple was as um when Michael had asked the question about what are the trade-offs if you're talking about kind of filtering out what what log traffic ultimately goes Downstream in this case to HLA and so when I started sending logs in from my cluster what I found was the different microservices were generating output generating logs in slightly different formats and so I determined probably the best way to go would be to eat the elephant one bite at a time so I picked one microservice to focus on for what I was going to show you today and so that service is called the front end so I just added a function to this pipeline through which the data would flow and I just used a built-in function that stream provides called drop which drops any log message which doesn't match the specified filter and so by applying this filter I'm avoiding sending traffic that's from one of the other micro services that I haven't adjusted and you know prepared HLA to ingest yet from showing up and and generating a bunch of because HLA what will happen is when it gets data it kind of decides oh this is the kind of data that's coming in I'll give it a default name and I will start collecting samples and learning from that data and so by adding this filter at the front of that pipeline I'm preventing that from happening before I'm ready so the other thing that I did just to demonstrate how is a powerful tool which can prepare your log payload so that when it gets to HLA there's a lot less that you have to do using native JavaScript and so what I did there was I just added a couple fields that would be sent added to the payload to generate a friendly severity key and time stamp key which HLA could basically just read from the get-go without having to write any parsing logic within the native JavaScript function that HLA uses to do that and it's really you know as many things in it there's multiple ways to do the same thing I just found that this was more straightforward to do because of the fact that I could flexibly collect sample payloads such as this one and then observe how my payload would behave interactively within kribble stream so here's an example I just I had a I had saved a paid a raw payload I can load it up and then this shows what it looks like coming in and then by clicking out I can see that because these don't fit the um the specified filter for the kubernetes label these packets would all get dropped and not get passed along to HOA so now that I've done my transformation put the data into the shape I want it to take on before it gets passed along to its ultimate destination that's when I configure my destinations within kribble stream so by default there's a Dev null which just gives you an out so that Ulta you know the um the trail that your data follows by default it ends with this devno which just kind of sends it to the bit bucket and that's really just to give anything that doesn't specifically get caught by your filters that you've put in place some somewhere to go so it doesn't end up you know looping or uh it also gives you flexibility where you can make that fall back something else such as just dump it all to an S3 bucket for example yeah I've created three TCP Json destinations which correspond to health log analytics data inputs HLA accepts a number of different protocols this was just I chose this one because it's fairly low overhead we also include you know we can listen for various other protocols like elastic um there's a rest endpoint as well which takes a more kind of rigid rest API type interaction I found the TCP data input to be really lightweight and easy to work with when connecting to so that's the one that I've used and just for simplicity's sake I've got a few different ones just to give me a little more flexibility with setting up how they get handled once they get into the HLA side and again there's constant transparency transparency where uh just like with sources on a destination you can kind of Snoop in on live data and see what that looks like you can save it off as a sample file which you can then you know use to further adjust and curate how your Transformations are taking place within the tool and then the other destination that I set up is an S3 destination one of the challenges that we get sometimes when we talk with customers about HLA is they're looking to um they're looking to kind of revamp how they handle logs and they like the analytical aspects of HLA but they still need to solve for things like long-term retention archiving ad hoc querying and fribble can help with that because we can have our data send not just to HLA but also to some type of archival storage I'm using an F3 bucket but has native support for Azure Google Cloud as well as a number of other end points that can ingest and retain log data and good one to call out here is uh minil so minio is an on-prem uh I'll say S3 storage emulator lets you basically send to an S3 API but it's a on-premise emulator essentially nice yeah so it acts the exact same way as Google Cloud Storage or Amazon S3 so in my case what I did once I had my log traffic kind of bifurcated going to HLA as well as S3 was I just set up a quick proof of concept uh Athena database Athena is Amazon's tool that lets you use SQL to query data that's in S3 so here's my raw logs and I can just run this example SQL query against my pod logs and dump out really whatever specific information I'm looking for from the logs even if it's not something that I was kind of that I knew that I was interested in when the log was being generated and there we go so that kind of covers how I've getting my logs from my cluster through cribble sent off 2s3 as well as HLA now on the HLA side what that looks like is data coming in Via what's called a data input data input records are what instantiate a listener and per that diagram that Mike had that show the architecture The Listener is running on your mid-server and so I've got some different endpoints that I'm listening on three of them are active I've got one for my control plane one for the Pod logs and I've got one that's taking my playback from my S3 bucket when I trigger that and so just a quick look at what a data input record looks like so this is where I tell it what kind of data input I'm using I'm using a TCP listener uh I give it a port to listen on I want to be super safe so I'm using TLS and then down below it says hey yeah I've got this guy here is sending me data and here's the uh the log lines per second that are going through so this is kind of the front door for for HLA once it hits that data input the next step in the in the path is called the data input mapping data input mapping is where we tell HLA how to determine what service within your cmdb is being reported on and so there's a couple couple things to just call out here one is the fact that HLA is sampling your data and that's you know gives you an inter a way to interactively kind of tweak how you're parsing the data and see what the result is prior to hitting that publish button and kind of committing it into the system and telling the AI okay you can start observing this data and determining what's what's uh my Baseline what's considered normal and and Beyond the alert for When Things fall out go outside the bounds and that can be seen in this middle section here where it basically takes your samples and runs it through your JavaScript code whatever JavaScript code that you've put in place to kind of guide it to tell it what application service this message corresponds to and then further break it down into a component of that service and then what's called a source type and the source type is kind of the next step in our journey Source type in health log analytics speak means file format and that's really where you make sure that health log Analytics understands how to pull out the important nuggets from a given piece of log data and so here's an example and what health log analytics does is it ingests your payload and again you've got samples that you can draw from to pick exemplars and make sure that it's working correctly against the any kind of variance that you have in the logging that's coming through that particular service and then down below are all the keys that it's pulling out and then all you have to do is just make sure that the keys that are coming out of your payload are tags appropriately and health log analytics is really looking for a couple key tags it's looking for a severity tag a time stamp tag and a message tag those are kind of the three bare minimums and then there's some additional tags that you can assign you can assign a host tag that's generally something that you want to do because that's what helps tie it back to a CI in your cmdb and the result is you get alerts like these so this is uh application service that I defined in my instance corresponding to the boutique application made up of multiple microservices running on my cluster and then because I've been firing you know the Space Invaders have been getting blown up left and right I've got pods dying and Chaos ensuing inside the cluster and so as a result I get a bunch of alerts um where they're coming from a pod you'll see It'll refer to the specific pod that's being impacted here and so that's kind of the end-to-end journey um and just you know just to reiterate the how cribble makes it better is the ability to route and pipeline your data on the front end before it hits HLA means that your log your your log sources your endpoints that are generating log data they just need to be set up once to send their data to and then you can adjust and slice and dice how the data looks and where the data goes without having to involve application teams and repoint things and deal with you know kind of that very low level granular um change within your environment and the other call out is just I I found it very very easy to get the data to look how I wanted it to by funneling it through cribble first just because of the fact that it's as simple as just hitting that live data Tab and you can see the data at any of the points along the way and see what it looks like it just made it very easy to adjust things so by the time it hit HLA I had minimal minimal things I had to do any questions all right Mike if you're still there did you want to just touch on your um your breakthrough with uh the containerized mid and the um yep and the roll yeah definitely so um the key thing here is and we'll send this out as part of the slide deck uh when we get back um is when we get done is that the prior to getting this figured out it um in order to do uh discovery of AWS right the the best method and most recommended method from AWS standpoint was to use an instance profile right and then use assume role capabilities from that instance profile and it was great because then you didn't need to manage credentials in the servicenow platform and you don't have to you know pass around API keys and API tokens which most AWS customers are not wanting to mess with anymore for security reasons so um but at the same time we didn't have it documented really clearly on how to do an instance profile with a containerized mid server and so I documented that there there's the two links on this slide are to my GitHub repo and the community article both the exact same article just two different places if you want to go check it out bottom line you have to build out and attach an oidc provider to the cluster and then pass that I am role down into the cluster and then map it to a cluster role inside that cluster and then fire up the mid server as that cluster role so it it works quite well and and finally enables us to do containerized mid servers using credentialist Discovery for AWS so kind of a I think a big important thing that we needed to get hashed out so I'm glad we got that worked out um feel free to go check out the links and as always you know drop me a note or add a comment to the community article if you have questions I think that was it for the kind of demo portions and we're actually kind of just bumping up against time um I know some people may need to leave but for those that don't as always we open it up for questions and answers at this point anything that you want to talk to us about around kubernetes about servicenow whatever it may be Now's the Time feel free to jump off mute and if I think maybe let's stop the recording so if people want to ask questions they can feel free