all right welcome to session three of the cloud native uh beers with engineers uh we're super excited to uh get together and chat with you guys um we're gonna be talking today about cloud native operations which is a store app that you can get off of the innovation labs that dramatically eases the process of setting up kubernetes discovery we talked a bit about it yesterday or excuse me in our last session but we're going to go much much much deeper on it today a couple of quick points um avolet will introduce here his beer here shortly i am actually on a super strict no beer diet so unfortunately yeah i'm trying to get i'm trying to get drop a weight bracket for my next fight so there will be no beer for me today i will just enjoy you guys enjoying your frosty beverages will you want to give us a rundown on yours sure today i'm trying a wrench as in the tool uh hazy ipa by industrial arts brewing in beacon new york nice it's always fun that there's so many great little local breweries and uh um i've i i just recently was out in mancos colorado and had a a local cider from out in mencos and that was top notch nice um so let's go ahead and get things moving just formality really quickly um you know this is we're always talking about you know forward-looking statements here as you guys saw in the last couple of sessions we have a tendency to peel back the covers a little bit and show things um fairly deep and in sometimes in futures so don't make any purchasing decisions or stock decisions based on what we're showing today this is you know purely just here for us to all learn together and work together as we go forward so uh as again you know it's pretty informal this is our agenda it's gonna continue to stay this way until you guys tell us hey let's add this or do this or whatever but you know we'll just quickly cover you know why we're here who we are uh we're gonna dive into cloud native operations and then we'll have a q a and workshop session at the tail end so why are we here this is the same thing as always but i do believe it bears repeating right the whole goal is for everybody to come together and talk about where we're at in our cloud native journey the struggles that we're kind of trying to overcome while we're here some of the capabilities that we have and understand how we can better leverage the servicenow ecosystem in order to continue on our maturity path down the cloud native maturity model so who we are uh if you don't know me by now mike gallagher i am an itom architect um i have been in i.t for a very long time um if it has ones and zeros in it i've probably had a hand in it at some point in my career right now i'm heavily focused on kubernetes and all things cloud native and a side note about myself i spent a lot of time playing board games and training brazilian jiu jitsu and actually this my background today is from tap cancer out which is a cancer fundraiser jiu jitsu tournament that i'll be doing in early june so i'll i'll post a link on my linkedin if you would care to support me in that i would be greatly honored will will hallam i've been at servicenow for about a year uh in it for over two decades um currently focusing on automation and that's what's kind of brought me to the cloud native with servicenow world um public cloud is a great set of it's kind of like a big set of lego blocks for for it and servicenow is that a little mindstorm computer brain that they used to provide in the the lego robotics kit that would make the legos actually move together and and do cool stuff and so it's really been a lot of fun figuring out how to make the the tedious things that exist in the it world so much easier and more predictable with servicenow and in my spare time i like to hang out with my family play pickup hockey and video games awesome all right so we'll start off with kind of a high level overview of what cloud native operations is we talked about it in depth uh last week well at a high level last week last session and then today is going to be super deep dive so what is it right and this in italics here this is the blurb from the the store app page um which we can we can include a link in and send out to everybody but really ultimately at its core it's it's a solution to quickly deploy and maintain right the entire itom stack of capabilities within a kubernetes cluster and it it really dramatically streamlines discovery and health and service mapping and all of those capabilities as we're you know managing a kubernetes cluster so just at a high level benefit right i talked about it's you know highly streamlined much much easier to deploy and manage it actually makes some changes to the discovery patterns so that the kubernetes discovery patterns don't require a credential instead it utilizes a kubernetes service account that it assumes in order to do all of its discovery components and authorize against the api which is just flat out a better and more secure method it also changes the way that we can discover resources within the the kubernetes cluster instead of running patterns every every couple of minutes that actually uses an agent script um to run locally and ingest events from the cluster in 60 seconds or less so it's much faster and it's more streamlined and then the last piece is it has an automated deployment of the agent client collector and if you're not familiar with the agent client collector it is our kind of an all-in-one agent for visibility and health and log ingestion so we use the agents to discover the kubernetes nodes themselves we use the agent to ingest metrics and logs from the pods we use the agent to discover traffic patterns as traffic is communicated back and forth between the pods and services on the cluster and then make recommendations for service maps based on that and then we can also use it we don't have it configured in our demo today but we can use it for log ingestion for health log analytics and as always it's informal if you have question feel free to come off mute or raise your hand you know jump in if you have anything you want to add or ask so will you want to kind of talk us through the architecture here yes so um as michael get into when he kind of walks through the the install process the way this um kubernetes bundle is delivered from once you've installed the store app in your service now instance is there's a a deployment page which produces some one-liners and there's uh install and remove one liner for uh straight kubernetes and then one for uh openshift and what that one liner essentially does is it retrieves a template yaml file from your instance and then it runs a little prep script that prompts you for things like the mid-user the mid-user credentials and the uh the api key for the mid server websocket endpoint and it basically uses that to fill in the template and then it generates uh something like what i've got in this diagram here when you deploy it to your cluster and so what that looks like on your cluster so right here we've we've got our you know we've got a cluster up here and let's see if i can annotate so we've got our cluster here and then it for sake of this kind of uh example we've got four nodes and so within each node you end up with a pod that's part of a daemon set which hosts our agent client collector agent and that provides service now with visibility to what's going on inside each node for example you know what pods are hosted in it and then what container images are being run underneath each of those pods and then it also deploys a stateful set pod which houses a containerized mid server and uh an acc agent is in there as well to basically kind of keep track of what's going on with with that that mid inside the uh inside the pod and then you know all the standard kubernetes trappings are included along with that so if you know this mid starts to misbehave we just kill this pod and it'll stand up a new one all of the registration which um kind of out of the box is done somewhat manually as far as you know if you deploy a mid uh to a server a standard server uh you um generally would have some manual steps involved with installing the mid package and then validating that mid once it reaches out to the instance and that's all done automatically within this within this kubernetes bundle it also defines secrets with which to store the authentication tokens such as the mid user and password as well as the api key used by the websocket endpoint and in terms of communication i've kind of drawn that in with the green and red arrows so the green arrows represent communication between the agent client collector and the mid server that's running inside the stateful set pod and then the mid server is communicating via the you know indicated by that red arrow back to your servicenow instance just like any other mid server would make sense any any questions kind of about what the architecture looks like all right so i will go ahead and actually jump over to my instance and show you a little bit about oh i have to clear the annotations first i can i can do that all right um so once you actually here let me um start with the store so in the store this is the cloud native operations uh application itself um you'll go here and you'll see as of today it is only compatible with rome so if you've moved over to san diego already um hold hold on and i my hope is we'll have a san diego capable version soon um but we're not there yet so right now we have version 243 and the key thing is there's there's this user guide here is actually quite good on how to install it and set it up it talks through the creation of a user necessary that has some specific roles that it needs above and beyond what a normal mid server needs because this mid server is actually creating services on the fly for you as well as building up event management integrations on the fly for you so it's a little bit different from a standard mid server user there's also a couple of other components there are a couple of stateful sets that you can that need to be added to your instance one is absolutely required for it to work at all and the other is if only if you're using the health log analytics capability um the the staple set does some changes to the kubernetes patterns and a couple of um sorry the update set does some changes to the kubernetes patterns in order to remove the credentials requirements and a few other components so once you've got that process done and you've got it requested and installed in your instance you'll have this new cloud native admin section here and you'll go to this deploy to kate's cluster here and you'll see this is really just a shell script essentially right that is going to pull down the config off of your instance and at the end you know save it to a local yaml file and then it'll pull down an actual shell script called sn app deploy to run through and ask you those very common questions those configuration questions and um here we can just quickly do it i won't actually complete the deploy because i've already got it up and running in my cluster but it asks you hey what instance do i want to connect to you can say you know then this is the username that the mid server will log into the instance as so for now again since we're just doing testing i'll do that oh and it's going to fail because it actually reaches out and validates hey is this username and password correct and accurate and does it have all the right roles so we won't try that we won't go through that the completion of that but once it's there the next thing it's going to do is ask you for your api key which it gives you a link to go to your instance and copy the api key down and and then go once it does that it will oh it also asks you what namespace you would like resources deployed into on your instance so i put them in the servicenow namespace so we can see here are the pods that are currently running in this namespace looks like i've got one acc agent that it's trying to restart but all the rest of the acc agents are up and running and you'll notice the mid server is running and two of two is because it's got its sidecar mid server as well so now that's up and running we can actually go back to the instance and we can go to there's this new cates mid servers right so from here these are the kubernetes mid servers that have been deployed using the cloud native ops toolkit and um this is the the one that we're running on here and you can see it's up and it's running it comes up automatically validated it comes up very quickly and it comes up with the um the agent client collector web interface already up and configured on it and so at the same time it also deploys agent client collector agents as those as that daemon set across each of the nodes in your cluster and so this these are those kate's agents that are deployed and you can see here you know here's where it's collecting data it looks like this is the one that was having issues we'll go ahead and see if we can get that to restart just quickly uh resume data collection and then as those are up and running now they'll start ingesting metrics and traffic from that cluster so if we go look at the kubernetes dashboard so this is the cluster that it's currently deployed in it's got six name spaces in there and just for example purposes i deployed the the boutique app which is the microservices demo from google in the default namespace just so for testing purposes and so we can see some traffic and see how one of the things that are going on so um these are all of the services and pods that make up that application you can see actually some of them have incidents already against them and we can go and dig into that here momentarily from there we are actually ingesting metrics from each of these components already and we also can do tag-based service maps so one of the things that it also shows here in the cloud native is this service map candidates right so by default it filters out cluster ip services because generally cluster ip services are only available within the kubernetes cluster and they are not externally facing services but in this particular case since we're not actually managing a public application most of my services are deployed with cluster ips so i'll turn off that filter so i can see all of the potential tag based service map candidates so each of these are our services that it's recommending be built based on traffic patterns and tags and you can see here's the the cluster id that it came from so i'm primarily working on cluster id 616 right now so for instance if i wanted to say okay i want to actually create one from this product catalog service i can click on that and i can say okay here's all the components okay that makes sense i'll go ahead and click this create application service button out of it and that'll actually create an application service for me and and automatically build out the service map based on the tags that are ingested from those cis this is a very simple straightforward service it's it's a kubernetes service here on the front end and that is backed by a single pod called the product catalog service pod right this is really a micro service and all of those you know multiple micro services that get deployed in that micro service demo app all roll up into what we call the boutique shop and so just to make uh that more kind of more visible and um make a little bit more sense i build a a tag based service out of the entire service called boutique and i did that if we go look really quickly here if we look at the manifest for the microservices demo that i deployed i added this label for each of the components called text service is the key and boutique is the value right that way and this is something that would should be done as a standard practice across all development cycles right is really every resource should have a standard labeling convention and be managed via labels there's one yes we'll absolutely record this and we'll ensure that we post it we actually have a youtube playlist for all of these recordings and we'll make sure we send that out with a follow-up email after we wrap up mike i'll throw the uh link in chat too oh perfect thank you you're welcome so so i've gone through and added those labels so that way we can kind of group all of those micro services into you know in my nomenclature a technical service called boutique right so now i've then built a service map based on that it's really straightforward you go in and say hey tag category is tech service right and then build out that tag tag-based service map based on those tag categories we can dive into tag-based service mapping if we really want to um but for now suffice to say let's actually look at the actual map of those services so this is the boutique shop and i'll um can i minimize that that's odd my sidebar there should be an arrow yeah it's missing down here oh there it is it was just super hit there you go all right there we go so so from here you can see here's our boutique shop and here are the various services on the front end we've got an internal front end and an external front end we've got our checkout service email service and each of those services is backed generally by one or more or more pods and in in one particular case we've got this load generator pod that's part of the micro services demo that does not have a front-end service because it's it's not intended to ever be accessed by anything else its whole job is to just run around and send traffic to everything else in the microservices demo so that we can see how things are are operating so and and i'll show you so this is actually that online boutique shop right i can kind of click around here and i could say sure i'll buy a tank top and uh you know why not i'll throw in a candle holder while i'm at it right so i add both of those to the cart i'll go ahead and place my fake order hooray my order has been placed good stuff so that all of that was all done and supported by all of these various micro services in the background and if we want we can actually go into kind of the service monitoring section here and we can look at the metrics that are being ingested from any of these various components so if i go click on metrics here and let's see here sure let's go look at the shipping service and i can say okay right here's my http operations per second right it's hovering pretty steadily that's mostly because of that load generator pod that's in there it's just kind of randomly banging around on that website and making sure that these graphs aren't zero but we can go through and we can look at any of those metrics we can look at the insights explorer from metrics and you know we can map all of these components in um we in this particular case um we can we're using prometheus on the kubernetes cluster to actually scrape those metrics and then ingest the metrics from prometheus but there are other various ways to do that that the agent client collector supports as well if you're not using prometheus so that's kind of a high level around kind of the deployment um the discovery process and um a tag-based service map and some of the ingestions but let's take a look at maybe what's going on on the operator workspace and see if there are some incidents that are going on it looks like i actually do have one because that's now red but let's take a look so my boutique app as it reloads here it's currently red with the severity critical all right and we've got multiple alerts that are associated to this and it looks like um i've got a particular node in my aks cluster that's having some resource constraints right which is actually causing some of the pods to not be scheduled properly right 005 nodes available one has insufficient cpu and four nodes didn't match the pods node affinity selector so it couldn't get scheduled properly um and that's just simply because of the way that service is set up we could go in and tweak it if we wanted to but it's actually nice to see some break right so we can see the events get generated we can see the incidents being happened and we can actually go in and if we want kind of kick something off one thing i thought i might do really quickly is let's do this let's say hey this is my production version of this particular application but i actually want to deploy a dev or a test version right so let's just make a copy of that manifest and we'll call it manifests um dash test.aml right and then let's just go ahead and change that one label probably just do it easiest with a said tech service boutique and we'll replace it with text service boutique dash test okay and globally on every line and kubernetes uh of course it's going to be difficult that's fine what did i miss oh different command on a mac that's fine don't worry about it i'll i'll figure that out another time um i'm too used to doing that actually in a linux pod um let me try that again man too many things going on here all right curse you bash history yup right so let's just do this boutique okay done and then let's create a new namespace really quickly okay actually put it in the namespace of boutique test okay so it's going through now and deploying that boutique app uh in that boutique test namespace and now they're all labeled with text service boutique test instead of boutique so let's go ahead and check our pod deployment okay i'll watch and wait for those all to come up so just to point out one thing um for those of you who have done kind of the traditional pattern based discovery of kubernetes there's the documentation tells you to set up two different schedules one uh to run the generic or the the cluster level kubernetes discovery and then one that runs more often that runs a pattern called kubernetes events when you're using cloud native ops the kubernetes discovery pattern still runs on a daily basis as kind of normal but in in the place of the periodically launched kubernetes event schedule there's actually a agent client collector check basically a script that executes on the agent client collector that's associated with the mid server in that stateful set pod which performs the same actions as the event-based pattern with the difference being it's more efficient because it's running from within the cluster itself and just pushing those events out versus pulling the cluster from uh an external mid server to try and pull those events out so it kind of gives you a a more responsive and accurate cmdb because it's pulling those events from within the cluster and pushing them into your cmdb i think it's set up by default every five minutes is that what we saw when we looked at the policy earlier today every 60 seconds 60 seconds okay yep so it's pretty responsive in its ability to detect changes to the cluster yep yeah so if i look for kate's here yeah so here's this this kate's events check that it's got configured and it's registered to both of my uh mid server agents and um where did i see that timing oh hang on on the check instance tab i'm trying to remember as well where we saw that there it is yep you're correct here it is every 60 seconds so it goes fairly quickly so now if i go back to my kubernetes dashboard again i now have seven namespaces instead of six and here's my boutique test namespace and all of those pods that were just deployed and let's go back to cloud native and look at service map candidates oh right so these are all going to be duplicates i actually need to go and go to tag based because i specified this tag based instead of traffic based which is what that's looking for it'll fall under this mike's apps so let's look at view candidates oh actually let's recalculate update the candidates and here we go there was boutique tests so now i can go quickly recalculate it hot is going on with my uh that's okay all right so let's go back to tag base services boutique test there we go list out those ci's there we go there's our pods here's our cart service right here's our services let's oh actually that's not what i wanted to do i wanted to actually go and view the map right i have a slightly different depth configured here which is why it's showing slightly differently than the last one but this is the same effective data right we still have the pods we still have the services it's all there another interesting thing to point out that we didn't talk about earlier was the track configuration files so i have i've got this configured to actually pull in the yaml files that are being used to define and instantiate these this one has not pulled it in yet so let's let me go back to the boutique service because i know i saw them on the boutique service man yeah the the yaml um i'm pretty sure the ammo extension only excuse me only runs when the main kubernetes pattern runs got it so um it was by default it's only going to get picked up once a day right so we can see if i click on one of these we can actually go down and look at here is the the json that's ingested that has the yaml file for that service and because that's a track configuration file we will store that and we'll store changes to that so if we were to go and make a change and then rerun the the total um kubernetes pattern the complete pattern instead of the event driven pattern we would actually see that get ingested and we would actually be able to track those those differences so we've walked through the deployment we've walked through the discovery process we've walked through metrics ingestion we've walked through kind of alerts and incident creation based on that will anything i've missed you think we should touch on um i don't think so i think that's a pretty good overview like mike said you know shout out anytime or put in the chat if you have any questions um one thing that i mean since since we've got plenty of time um one thing that's probably worth talking about is how readily this kind of thing could be incorporated into like an automated um cluster you know deployment ci cd pipeline that kind of thing since kind of out of the box what the instructions tell you to do is you know pull down that one liner and then execute that and respond to prompts yep so um actually if i can grab the screen for a second i can just kind of show how that could pretty readily be incorporated into a ci cd pipeline okay you see my screen okay yep terminal window yep cool so down at the bottom here just kind of pasted in that that one liner as delivered by the instance and so what i did to kind of create my own customized version was um two things i basically just deconstructed this command to initially i just broke down broke it down into the two component parts so i took the first curl command that's in the one liner and just executed that and what that gives you is a copy of the template that is downloaded from the instance so normally this is downloaded it's processed and then it's removed cleaned up by the one liner so in order to kind of you know inject your own customizations all you have to do is just make a copy or save a copy of this then the second thing i did was went back to that one liner dash history so i went back to the original one liner and just deleted the front part like this and then i just redirected that to something like this that basically pulls down the install square the the kind of the uh the formatting script that we run and once i had a copy of that i just went in there and just modified the end so it's basically you know just prompts for those various credentials that are required and the instance name allows you to do things like configure a proxy if you want the mid to communicate with the instance over a proxy and then at the end it runs it would normally run you know cue cuddle apply and then it would clean up after itself so i just kind of invalidated those last two commands and then executed the script by hand and this produces the same prompts that mike went through earlier and ultimately generates it basically just updates this template file so that it contains you know fills in the kind of fills in the blanks with the values that you provide via the input so once that file is created i've now got a kind of a standalone file that i can execute multiple times over to push out a uh standardized cloud native ops configuration without having to go through the same process over and over and that kubernetes manifests can be you know propagated to any number of different clusters so what that looks like is something like this let's see [Music] here's yours [Music] oh it looks like it was too smart i'm good it cleaned up after me all right so i'll have to actually go through the full exercise here okay thank you okay so [Music] in order to as mike mentioned in order to get it to complete you have to kind of give it valid credentials because otherwise it won't successfully verify it's able to connect your instance so retrieve my api key for my instance so for example if you're going to automate this you would probably just put those values immediately into some kind of a credential store and then just instruct your kubernetes cluster to pull them from there versus putting them directly into your deployment but really what's nice is once you've got your um once you've got this this basic file you can basically take this starting point and then customize it as needed for your specific implementation and it gets you out of having to manually kind of run through that that scripting activity every time uh an example we ran into was mike was having an issue with some uh the default resource allocations that were being provided to the acc pods and one of the ways to fix that would be to customize the agent client collector configuration file that's part of this image and a way to do that would be to create a customized image by pulling down you know this base image making the requisite modifications and then pushing up a customized image to your own container repository and then you would just go into this manifest and modify that container path to reflect your own you know private repository as opposed to pulling the one from the public ecr which is what the the tool does out of the box it also provides you know also gives you the ability to apply additional tagging if you so desire um you know really basic anything that you would normally do with any other you know kubernetes workload definition so it's pretty straightforward and lends itself to automation pretty easily yeah great question jim jim asked in the chat if this is ready for aks eks and gke so um the i have a customer right now who's actually going through and doing this process and added into their azure devops pipelines um that actually goes through and and builds out the whole manifest and pushes it into their gke clusters um i have deployed this on aks eks and gke and i have not had any problems um with any of those there's a there's a couple of things that i would say are oddities um like we're using the agent client collector to um do things like look at free disk space on the kubernetes nodes which makes sense on uh you know on a cluster that you're managing but on an aks cluster that's really up to azure to ensure that your your node has enough free disk space right um that's not generally something that you need to be too terribly concerned with um there's a there was one other out of the oh um in general azure aks does not have a um it does not expose the cube controller manager um and that's something that we look for as part of the standard pattern and so therefore it kind of squawks about that when you try and discover that but that's not a deal breaker and is generally just kind of a minor annoyance and is not also is not unique to cloud native ops however here is the part where we have an ask of you guys um if this sounds interesting um please go to the idea portal and say hey we really would like to have cloud native ops push to ga as it stands right now it's set for ga and q4 of this year and i think it would be beneficial for lots of folks to have that pushed earlier and get more kind of development focus on it so um that would be my ask i have actually one other ask really quickly and i'm gonna shift gears to talk about it and and that is um around um having an immutable container i will share this idea out but this is from a customer who is using our container docker file that we provide and deploying it and their security team is rightfully saying hey this container image doesn't follow kubernetes container best practices which is really that your container should have a read only root file system and then you can mount kind of a staple storage or some kind of other storage if you need to have things that can change but but you're not a really best practices you're not really allowed to change the root file system of a container and the problem with that is as it stands today the mid server code very much is used to running on i am on a vm or i am on a server and so i have the ability to write anywhere and everywhere and so as a result it assumes that the container that it's running in is in in read write mode and so they have put in this idea to have us focus some development efforts on um developing an and an immutable container for a mid server that would have the ability to move all of its read write functionality to a specific directory or a specific location that can be done and defined in a safe and secure way in kubernetes so i'll throw the link to that in the chat if you guys wouldn't mind throwing a few upvotes on that that would be great because then we can get some more development efforts focused on that