all right excellent off sounds good everybody Welcome um thanks for joining beers with Engineers sessions 20 right it's amazing to think we've had 20 of these sessions now um really excited to be here with you all today will and I always joke that this is our our favorite work day of the month so appreciate you jumping on and joining with us today um we'll just kind of review um how we always approach things um first and foremost there's always the Safe Harbor notice um you know things that we'll be talking about today may or may not be released yet um they may or may not come to fruition exactly as we have talk about here today from a feature standpoint so don't make any purchasing decisions based on these conversations today and uh we just want to make sure that everybody's clear and covered so as you guys know keep things pretty light here so um genda is fairly straightforward we'll talk about why we're here who we are kind of run through the tech Deep dive and then open it up as always for Q&A afterwards so why did Will and I start this right the the whole goal of this was to connect with people in the service now Community who are also on the cloud native journey and bring folks together and talk about relevant technical topics that are important to to that group of folks and to build a community about it so um it's really important that we build this community and talk through all these components and and we're just so blessed that we have so many great people who join us every month so uh who we are I'm Mike gher I am the uh manager of the Enterprise applications team here at at drw um I have been doing technology for multiple decades um I I love um in particular it operations management that's been a long focus of mine um I am at kubernetes nerd and run most of my home lab stuff on kubernetes um and I I one of my great philosophies is that we need to solve business problems effectively efficiently and intelligently with technology so um and today I am not uh not drinking a beer because I'm feeling a bit under the weather so I'm going to have a liquid death mango chainsaw so will over to you nice like the name yeah uh Hey everybody will hallum uh itom architect over here at service now uh been working in technology for several decades right now focusing on it operations management particularly about automating tasks which tend to be mundane or or repetitive within the it operations management space with a particular focus on cloud native public Cloud a lot of kubernetes work really enjoy being able to make those make those things painless seamless and automated and today I will be enjoying from a a local Brewery in my area a lake George's IPA from Adera Brewery sounds good yes okay so today we're going to dive into something that came out earlier in November called the uh it's it's under the uh ACC visibility um plugin and it's called Cloud native operations for visibility and it features a functionality called the kubernetes Informer so it was released uh November 2nd as part of ACC V 3.4.0 and so some highlights of what this provides the first thing it provides is a turnkey installation we now have a public container image that delivers this functionality so for those of you who are familiar with prec iterations of cloud native Ops you probably recall that after releasing that for a a having that be out for a couple of months we um kind of pivoted from providing public images for those uh kubernetes or containerized workloads towards just providing a a recipe and so and that was due to some inherent vulnerabilities on the underlying distribution that those images were based and uh just an overall very high level of sensitivity to vulner vulnerabilities that get flaged container images and so that introduced some additional rigor on the customer side that was not in all cases um well-received and so in an effort to make the customer effort less when it comes to deploying service Now functionality in their container ecosystems we now have the Informer with a publicly available image it can be there are instructions in the documentation so it can be pulled in and hosted in an internal repo as many customers are apt to do I think Mike that's kind of your standard practice at drw is you don't get stuff from public repos you put it all through your own internal repo even if it's just you know caching those uh those publicly available images correct um in terms of deploying those to your cluster we provide Helm chart and yaml manif off the shelf that you can just use to deploy or customize and then deploy into your kubernetes clusters and so what does this capability do it provides quick cmdb updates pretty much the fastest most responsive updates to your cmdb based on changes within your cluster that you can obtain yeah and that's because it's extremely lightweight it's constantly running and interrogating what's going on in your cluster and when it detects changes it sends those in directly to your service now instance this does not use does not require cannot use a mid server it communicates directly with the service now platform that it's configured to um pair up with sends those events in Via a dedicated event ceue those events are handled via dedicated event processors with your n instance and they update the cmdb in near real time anywhere from you know moderately sized cluster it's a matter of minutes and we'll talk about the um talk a little bit about scaling uh in a little bit but the uh the responsiveness is the best of any of the options that service now provides for discovering kubernetes so I touched a little bit on the resources within the instance side and that's pictured here on the right side of this diagram which kind of shows how the Informer is set up so on the left side you see the uh your representation of how it looks in your kubernetes cluster and the Informer itself continually pulls for events change events within your cluster pods created destroyed um various workloads staple sets Damon sets created destroyed uh does that by querying the kubernetes apis directly then when it detects changes It prepares those changes into a payload and it puts them in a queue and then periodically uploads the payloads from that queue to the ECC Q on your service now instance from there the events do reside in a dedicated event q channel within the eccq and they are pulled by a dedicated event processor and then there's a couple different Gates that are in place to prevent uh highly active kuber cluster from flooding your instance with uh with traffic and impacting the performance of other instance functions uh one of those things is we limit the rate of the data sent into the instance to uh no greater than a megabyte every 30 seconds and there's an internal CU on the Informer side that will hold up to a 100 Megs of payloads and just kind of releasee those in one megabyte chunks as applicable the Informer has built in back off so if your instance is being patched or is otherwise non-responsive the instance isn't going to keep hammering your network trying to reach transmit it has a progressive back off where it just keeps increasing the the um the weight time before the next attempt to connect and then once the payloads get to your instance they do go to a dedicated event queue and the event processing is limited to occupy a maximum of two concurrent threads on your worker nodes that are execut exting you know all the code that executes on on your instance so this uh processing can't grow without bound to the point that it affects resources available to other instance functions so and what's interesting about that is those you know one Meg and 100 Meg limits seem kind of small um but when you stop and think about the fact that most these updates are going to be in the kilobytes size right um that's a that would be a significant amount of changes on a cluster um in a very rapid time frame to be able to like kind of flood those buffers so um it's a it's a pretty reasonable buffer size and a shouldn't really impact the performance of your cluster um which they did some testing on and B um are should hold quite a few updates before it overruns those buffers and starts you know tossing stuff on the floor yeah absolutely and I was um I was curious so I was able to track down internally within service now um Bubble I was able to get a look at the source code the go source code for this um for the Informer and one of the things that I noticed is that and I think it's mentioned in the documentation but just more of like a side note so I didn't really notice until I was looking at the code and some of the comments it actually has builtin um kind of noise built kind of a built-in squelch function where if a given event is happening more than 15 times a second then it will by default kind of suppress that event because there are some things that happen within a cluster that are just so so rapid it doesn't make any sense to to try and capture those so when it's happening multiple times a second there's actually code in there which I thought was pretty clever that they're automatically kind of squelching that kind of stuff and focusing on the things that are more you know along the lines of like something that lives for minutes or hours as opposed to something that lives for fraction of fractions of a second yeah that makes sense yeah yeah it's pretty sweet so um a question that we've been getting for the last year two years and is not going away anytime soon because we've got all these different options for discovering kubernetes is you know when do I use which option what's the best practice which one is the best um and so as with many things on the service now platform the the general answer is it depends and so what I try to do here is capture some of the things that determine whether kubernetes inform might be the you know an attractive option for a given customer in contrast with some of the other me mechanisms that are available yeah so the first thing is I mentioned this is released as a publicly available image you can grab this image off dockerhub cach it internally if you know if if you want you can pull it direct there's you know it's it's it's hosted on dockerhub like any of the other publicly aail able images it's very lightweight and it scans with zero vulnerabilities and so if you appreciated some of the benefits of the original release of cloud native Ops but didn't like the rigor of having to manage the M generate images or the fact that the images were fairly large they are based those both the agent and the mid server images that came with that first iteration are based on full distributions full OS distribution so they tend to be kind of large and that was also what led to them having a bunch of vulnerability hits is because there's a loads there's loads of packages in there this is very Bare Bones it's basically it there's a basic goang um build container that's used to compile it and then there's literally like this very basic shell of a container image that's used to copy in the executable I have a couple file systems and and pretty much that's it so if that is appealing then kubernetes Informer maybe maybe right for you uh there's no Discovery schedules whatsoever required even the original CNO 1.0 or first iteration of CNO did create a cluster wide schedule to run uh by default once a day when you deployed um Cloud native Ops into your cluster if you'd rather have zero Discovery schedules and just have your clusters discover themselves and send the data into your instance then comm's Informer may be for you if you'd rather not manage cluster credentials the Informer may be for you similar to the first iteration of cloud native Ops there's no cluster credentials required all the permissions are provisioned via the helm chart or the Manifest they create a service account uh the permissions that that service account are given is completely transparent so you they are pretty minimal as far as providing just what's needed to begin with but if you know you did want to tweak something here or there in fact I think uh somebody had asked a question can we control what is discovered today that would be one method is if you tweak the service account so that it can't interrogate um kubernetes resources that you're not interested in then those resources wouldn't get pulled into the cdb number four if you want fast turn around like I said this is the fastest way get your cmdb updated to contain kubernetes information it and we did do some performance testing and published that in a support knowledge based article where we share kind of um small medium large T-shirt size clusters and change rates and the commensurate speed of updating the cmdb that comes with each of those so it's fairly well understood at least within different spots in the Spectrum what your expectation should be so you can review that and see whether that meets your needs in terms of how quickly would we like our cmdb to reflect what's going on in our cluster number five is is kind of a caveat which is is the kubernetes Informer the discovery that it performs is independent of our Discovery patterns which are used by our agent list server list Discovery as well as the original iteration of cloud native Ops so if you had started your kubernetes Discovery Journey using patterns and then you had customize those patterns perhaps to add some additional attributes pulling in from the metadata to discover some additional resource that was not discovered out of the box or if you're leveraging one of the um one of the extensions that comes out of the box with the kubernetes pattern then you would have to address how losing those additional bits would impact your kubernetes Discovery and then decide whether the juice is worth the squeeze for getting that low administrative overhead and and the uh and the very fast update speed against those customizations that you'd performed within kubernetes patterns another point there is metrics ingestion right for health right this this is just purely visibility it doesn't ingest any metrics um also just a quick point I went and looked at the uh container image on dockerhub and the Informer is 22 makes right which um I think the old mid server that shipped with the CNO was like gigs yeah so yeah this is a massive like thin down um of you know the infrastructure components of it and it's also um going to be faster to deploy you know easier to manage so it it's just um in general if it if it fits for your use case it's a great option 100% And yeah that's a great call out so um even though this is under the cloud native Ops for visibility Banner or U kind of product family it does live alongside the previous Cloud native Ops plugin so Cloud native Ops when it first came out was its own plugin the kuber's Informer functionality is rolled into the ACC visibility plugin so you can run both if you you know if you've deployed CNO already and you're using some of those metrics and monitoring capabilities that also come with that iteration of cloud native Ops you can run them both use the Informer for Discovery and still you know if you're running some of the standard monitoring checks what have you with the original one they they can live together side by side that that not an issue so a few caveats both Mike and I have played around with this some in our various test environments and so just wanted to capture some of the things that like questions that arose that we got answered kind of up front so you don't have to reinvent the wheel um and then just kind of things to watch out for uh the first thing is basic off right now the only way for the um Informer to authenticate into your instance is via basic off and when I walk through the demo I'll show you how that gets set up it's U basically you create a username password record in the form of kubernetes secret and with a specific naming convention and then the Informer pulls those credentials and uses those to connect to your instance and populate the eccq yep there's you know road map items to add things like Mutual AU authentication down the road but right now it's basic oth only uh I already called out the fact it's self-contained it doesn't use patterns or any pattern extensions so if you're leveraging customization on the kubernetes pattern or one or more of the extensions that come with the pattern and discover extra things like the Amo files or um some of the specific uh open shift stuff then that you you can't just kind of Port that over to the um to the Informer I when I was looking at the Informer source code I did notice that it does have code to handle open shift so it's you should see successful Discovery within uh open shift environments it's just that open shift specific extension that's part of the pattern is not ingested or used by the Informer the Informer is a completely separate Standalone code Base today y uh already mentioned it doesn't use the mid you can't there's no option to use the mid today so it does need to have communication with your service now instance it does support setting up a proxy so it doesn't have to be direct egress but it does need a path by which it can communicate with your service now instance and uh already called out it it coexists with the prior CNO plugin so you can in theory run both although after you go through and install Informer unless you really hung up on some of that functionality like the monitoring stuff that came with the earlier Cloud native Ops uh I can see you just like totally throwing the original one in the dumpster because it's just so much easier to deploy and manage than in maybe log ingestion right um I think log ingestion was there in the old CNO but not here um but either way if you're just looking for what's running on my clusters and I need to know fast this is definitely the way to go 100% And speaking of log ingestion we now have a pretty cool story to tell which we're going to talk about next month about ingesting cluster logs pod logs natively using open Telemetry and sending that log data directly into our Cloud observability solution so stay tuned for stay tuned for that that's a fairly new offering it's pretty cool we're going to have some fun with that in December okay so now um gonna tempt the gods of uh of live demos and actually just walk through the process of deploying kubernetes Informer and just show hopefully how uh how quick and easy it is so first of all I'll just jump into my demo instance and this is a outof the Box kubernetes dashboard that comes when you install the kubernetes uh patterns and you can see I've got zero kubernetes resources visible to this cluster I mean uh to this instance right now no clusters and Nam space nodes Etc zero kubernetes visibility so what I'll do is I'll just start with the first section in our standard documentation on CNO for visibility deployment and just walk through the steps uh so I've already ahead of time in sure I've got my requisite prerequisite plugins applied I've got my system with Cube cuddle on it and and Helm and all that good stuff and I've also created my user um on my service now instance and given it the discovery admin rol so I'm kind of picking up here on on step two the first uh which it's kind of the first interaction with the kubernetes cluster that's required and essentially it's just to create a nam space in my cluster for the Informer to live in so I will go ahead and do that actually I'll just do copy past just to okay so I've got my Nam space and then the second thing is once we got our namespace we create that secret that I mentioned so we store the um we store the credentials required for the Informer to Place those payloads in the eccq as a standard kubernetes secret so I will paste that line in and kind of fill in the blanks here and I've kind of stored my credentials into some files so I don't have to try and off youate them after the fact it for YouTube double check my instance name COV [Music] demo okay now I've got my secret created so that's all I need to do because I am going to go ahead and just use the publicly available images if you're pulling your the image as is common with um with customers there's instructions for how to do that in steps in step four and then step five if you are going to route your outbound traffic through a proxy there's just kind of an admonishment to make sure you've got the proxy host name and Port identified before you attempt to deploy deploy the helm chart or the or the Manifest so that's my PRX now I'll just go to the second section in the docs which talks about deploying the kubernetes Informer into your cluster so I'm gonna use Helm and we maintain the kind of the lookup which includes the latest Helm and raw manifest download links in this um support knowledge based article these things I'm linking to all these docss uh will be included on the Links Page in the PowerPoint deck that we send out as a followup from this session so you don't have to kind of take screenshots or anything like that it's also very nicely Linked In the docs directly so you can kind of just click click through as you're going through the docs for additional sporting documentation so I'll grab this this Helm install command paste that here and then start filling things in so the first thing working backwards is the the helm URL so I'll grab that Helm chart URL from here I'll just kind of keep going backwards I'm not using a proxy so I'll just clear that out and I'm not using private repost I'll clear that out so the cluster name parameter is used for a couple things one is if you've got clusters already in your cmdb then you want to make sure the cluster name that you're assigning matches so that it doesn't inadvertently create a duplicate CI the other thing is it just allows you to apply a recognizable name to the cluster because otherwise what it'll do is the default name is something like Kate's cluster and then the big long kubernetes uu ID which is a little difficult to recognize visually so this just gives you the ability to apply kind of a cluster nickname or if there's a naming convention that you apply um so in my case I'm just gonna apply a fairly simple name to a cluster then instance instance name which I'll double check again cnov demo yep and then my name St name okay and we'll fire that down the pipe fingers crossed I've been really impressed with the the quality of these docks that they put together for this especially given that it's essentially a 1.0 for the Beret is Informer there's a lot of Health uh helpful links like these in here that talk about how you can you can configure the um memory limitation the larger your cluster is the larger the memory is going to need to be for the Informer and the performance testing knowledge based article which uh is also linked in our The Links Page talks about the guidelines as a calculation where you're based on the number of of PODS you apply a a corresponding memory limit memory request to the Informer pod and then um some helpful links on changing the discovery so by default a full Discovery gets run within the cluster by the Informer once a day and you can change that frequency to suit your operational needs it's it's really just um kind of a catchall that if something you know if something happens like if the Pod dies and a bunch of other changes occur you want to rerun that Baseline periodically just to kind of level set for any Deltas that might have been missed excuse me and then we also provide the ability to kind of go completely the other way and just do a periodic full cluster Discovery and disable the continuous discovery functionality if that's something that you would like to do okay let's see how my Helm chart ended up looks successful so we'll take a look at the uh pods on my cluster and we should have an Informer pod which we do this guy right here and so now if we want to see how it's doing we can pull the logs [Music] it's [Music] interesting trying to use auto complete and it's not happy for some reason because there's no get it's just just logs thank you by the way if you haven't played with it I recommend Stern for looking at logs from kubernetes pods you just give it you say Stern and then a a name space and then kind of a pod name like a fuzzy name and it'll stream all of the logs for all of the pods that sort of match that name to yeah that sounds awesome awesome yeah it's nice best things about kubernetes is like you you think man I wish there was a tool that would let me do this with kubernetes and chances are there is at least like yeah three of them it's yeah a great ecosystem to work in love it all right so this all looks happy we can kind of see initial Discovery completed sent ECC message so now if I go back to my kubernetes dashboard and do a little re load here bam look at all this kubernetes goodness that just came in within the span of minutes yep and as far as you know kubernetes Discovery this is fairly unparalleled and granted I'm on a smaller cluster but um within until you get really Large Scale based on that knowledge based article that documented the performance testing that was performed you get updates in your cdb within within minutes of the change occurring on your cluster which is it's just awesome so now so that's our Baseline and so now just to kind of do a quick demonstration of how quickly changes are recognized and and updated within the cmdb I'll just go to my pods listing and I'll pick a pod to get rid of and monitor to see how quickly it takes for it to be replaced so going go for the metric server pod all right so right now it says got a metric server pod running the identifier ends in Zed DSP I can kind of confirm that here so here's my metric server pod ending in Zed DSP so I'm going to go ahead and kill that not kill I don't know what I've got got subcommand uh too many of them illiteracy at the moment but uh okay so now I've gone ahead and deleted that pod so now if I repeat my get pod command I should see a new metric server pod running and I do so I've got metric server pod identifier ends in JV 69 and So within my test environment generally okay so got the new one has already shown up and actually the behavior right now the way um the way the Informer updates things is I'm just going to add install install status to my listing because I believe the way it at the way it behaves okay so yeah um so it's already basically updated that pod and under the covers what it does is it creates uh or it updates the record of the Pod that was deleted sets the install status to absent and then there's uh an hourly cleanup job that runs again the kubernetes tables and deletes any of the resources that are marked absent and so we can kind of see because by default when you're viewing kubernetes tables it's automatically going to filter out anything that is U has an install status of of absent and so now just a few scan couple of minutes since I performed the action we can see that the cdb is again up to-date reflecting reality on the cluster so in terms of so I mentioned there's a scalability dock out here on support and that does a good job of setting expectations as far as if your cluster size Falls within this range here's the update speed you should expect to see and and I kind of U I was very happy to see that because sometimes we can see that kind of performance expectation be more of an afterthought and we're left to kind of figure that out in the fields so it's nice to have a heads up on what to expect there so running through a few questions that we have in the Q&A um one of the question s is uh any opportunity to explore service mapping use case with the new iteration of CNO and so my understanding based on just a quick overview of the road map and the directional strategy for CNO in general is that it's going to converge more closely with the ACC visibility offering and as part of that I expect it is going to get more Rich connectivity data coming in from the cluster over time which will facilitate those service Maps um there's an effort right now to support service mapping via the agent client collector and so as um as the Informer intersects and converges with ACC I think we are going to see uh improve approved bread of data that's available from a kubernetes cluster for those who are familiar if you're running um kubernetes and you want to do service mapping your options today are either tag based or the open Telemetry service craft connector and the open the otel Service craft connector it needs instrumentation in order to do kind of a rich service map so there is that a bit of a heavier um requirement to make sure your cluster is configured in a certain way in order to get those those Rich service maps and so it does seem like dire actually by adding the Informer and eventually the the overall vision is for the need for a separate Informer to eventually go away in favor of a more general purpose ACC based capability but in the interest of providing some something that just gives you this massively quick update the with a with a lightweight footprint the Informer is kind of our first iteration on that overall strategy and and to be clear right like this it does ingest the kubernetes labels so that it can do tag based service Maps like today right now so if you're already doing tag based service Maps it'll ingest that data and update the maps automatically based on that data so um it it it's you know you can kind of crawl before you walk run and fly yep yeah you you have the same capability for generating service maps with this as you would with serverless Discovery or using the original iteration of CNO and that was actually another one of our Q&A questions was will it gather tags yes uh this is gathering this is going to gather all your um all your kubernetes labels if I pick a pod should be able to see um yeah here's here's our key values right here which are the kubernetes labels that are getting pulled in uh one comment from one of um one of my service now teammates was that uh he actually worked to direct IR ly with a customer whose specific needs directly impacted the creation of this and they had a need for uh you know Discovery and updates at a large scale and so that just kind of reinforces the fact that there was a lot of thought that went into being able to support those large scale installations where there's um you know clusters with thousands of PODS running and being able to keep cmdb in step with what was going on inside those very elastic envir requirements and let's see the last Q&A question had to do with service mapping and I think we kind of really already dug into that essentially it's tag based today um and the directional strategy is to ingest additional metadata to ultimately gather more insight into what's going on between individual workloads the challenge we've had so far is it's kind of that double-edged sort of kubernetes in that kubernetes is so modular you can have 10 different customers and they're each handling their kubernetes Network traffic using a different plug-in so it's a bit challenging to come up with a A one- siiz fits-all or a universal solution but we're getting there slowly but surely yeah that's really where the otel piece can come in and help out with that right if you're instrumenting at the code level then it doesn't really matter what um you know network provider plugin you're using um in your cluster um we can see all that all that data with Opel so it'll be that'll kind of help smooth some of that out absolutely okay well that's uh That's My Soup To Nuts down Mo I'm thankful that even though I was doing it live kind of without a net that it all uh it all worked really well really been impressed with this uh with this Informer just overall not hit any major hiccups or anything so far yeah that's really really pretty awesome the install was like butter right like um it's exactly what you expect from uh you know kubernetes platform um being able to deploy it quickly and easily with a Helm chart um utilizing Secrets um quite quite helpful um I haven't tested putting it into a get Ops style environment yet um but given the fact that it's a Helm chart I would not expect it to be terribly difficult so so I mentioned we'll have some handy links in the slide deck that we'll send out to everybody who signed up for today's session um it's another aspect of this roll out that's been really positive for me is lots of documentation augmented by lots of knowledge based articles and so I already mentioned our next session we're planning for uh December 14th to talk about the new capability within our Cloud observability product Suite to take logs and it's not not just kubernetes logs we'll probably kind of have that Focus since that's our overall Vibe here at beers with Engineers but um yeah it's uh and in one of the sessions Mike and I went to when we were at cucon this month they were talking about the fact that they've actually officially recognized and support pulling log traffic in through the open cementry collector I guess it's been there for a while but now it's finally like full GA 1.0 completely blessed by the open Telemetry collector project and and what that means is you basically just if you've got open Telemetry collector running which is available via Helm chart you just uncomment or you enable the log collection stanza in the parameters and then it's going to start pulling in logs and there's some additional you know um there's some additional settings required for pod logs because depending on your container runtime those pod logs are going to reside at a different path on your um cluster nodes but other than that that's kind of all you need to do and so hopefully we'll be able to kind of share some of that behind the scenes under the cover stuff with you guys uh next month looking forward to it as always thanks everyone for joining us um and look forward to seeing you guys next week next month yeah next month I wish it was next week next month yeah me too thanks everybody take care