welcome to session 10 of beers with Engineers the first one of 2023 we're really excited to have you as always we're pretty informal but we got to start off with the Safe Harbor slide um all we're saying is we might talk about stuff that's not yet been released or not be released in the exact same way as we talk about it today so you know don't go make any stock purchasing decisions based upon what we're talking about today um you know try to keep it to yourselves as much as possible but it's really all about how do we kind of share as a community right um so a real formal agenda same kind of thing as we always do um uh we're going to talk a little bit about why are we here remind ourselves of the goals of peers with Engineers um introduce ourselves to those of you who haven't met us before and then we'll Dive Right into Cloud native operations 3.0 the official GA version um and then we'll get into some q a afterwards so um so why are we here um for those of you who this might be your first time um welcome um the whole goal of this is to bring together servicenow customers and actually we should expand this to also say employees and partners because we get a lot of all of those folks but it's really all about customers that are in a similar place right and their Cloud native Journey so we can talk about how organizations are utilizing the servicenow platform to manage their Cloud native technology portfolio uh we like to get really nerdy on these calls and talk about nerdy stuff that's going to help solve specific business problems so pretty exciting so who we are if you haven't met us um uh Mike Gallagher I'm an advisory solution architect here on the it operations management team in the commercial space um I have been in it for a whole bunch of years um I have been a kubernetes nerd for you know the last probably four years um really super enjoying it and I also really like you know making life easier for everyone through things like Automation and you know centralized data which is kind of why I'm here at servicenow um as a side thing I trained Brazilian Jiu Jitsu um not right now was covid um but um that's a big thing for me and I play a lot of board games um we'll I'll let you go ahead and introduce yourself are you are you partaking of a beverage today Mike given your your delicate condition I forgot to mention as always I'm gonna drink a uh sweet baby Jesus which is a chocolate peanut butter reporter from dewclub brewing in Baltimore Maryland that sounds delicious hi everybody I'm will Hallam I'm an itom architect here at servicenow my background is in it operations infrastructure and lately been doing a lot um with automation especially in cloud or or hybrid environments in my spare time I love to hang out with my family I play a little bit of pickup hockey and uh probably more video games than I should today I will be drinking uh the Paradox breweries Beaver bite IPA and Paradox is located somewhere in the Adirondacks sounds delicious so um what we'll do excuse me we'll let you go ahead and sort of kick off the tech Deep dive and then we'll we'll jump back in and do some other components in a moment I'm going to stop Sharon will feel free to take over okay okay so it looks like I I think I got through making sure nobody was um kind of blocked from coming off mute so as always if anybody has any questions feel for comments feel free to just uh unmute yourself um you can use the you know the chat in the Q a as well if you're more comfortable doing that but but don't be shy if if you would like to just kind of speak up as if we were actually in the same room Okay so all right so Cloud native operations kind of the the too long didn't read of cloud native operations is it provides maximum kubernetes utility within the the platform with minimum admin overhead so specifically there's kind of two pieces from a uh from a productization perspective we talk about you know in the documentation and in some of the marketing materials we refer to Cloud native operations or CNO for visibility and CNO for health and and so the the high points of each piece of that are in the visibility side it means TurnKey cluster Discovery no more having to establish credentials in your you know your credential table and manually set up not one but two individual Discovery schedules for each cluster and um you know this applies this also has a slight Edge even over our automated um automated cloud based API driven kubernetes Discovery because it's got some more Dynam more ability to react dynamically to changes with in your cluster and so that's kind of the second major bullet under visibility event driven cmdb updates so that just means if you've got a cluster that's very elastic and it's uh killing pods and spinning up new pods with Cloud native operations that information is reflected in your cmdb in near real time on the health side um what CNO brings is turn again that TurnKey is kind of a repetitive theme because this runs inside your kubernetes cluster and pushes all the data out you don't have to do any of that prior setup with Bearer tokens or um establishing mid-server roles if you're running in AWS the the setup is is truly quite simple and actually I'm going to show kind of a just a part of the possible kind of example of that in in a couple minutes and then um so with that TurnKey health component what you get is an embedded agent client collector which will then send various events that are occurring within your control plane um right into the event management subsystem within servicenow it also collects a bunch of metrics both at the cluster level the Pod and container level and pushes those into your instance as well from support perspective it'll run on vanilla kubernetes if you roll your own uh or eks gke AKs and it does have hooks in it that recognize the the particulars of of openshift and so I mentioned that you know it provides a much more streamlined flow to the setup of you know making your platform aware of your kubernetes clusters and so as a way to kind of illustrate that I've prepared an example that I'm going to kind of kick off um here whoops I didn't mean to stop sharing I meant to minimize my get my share back I just meant to shut off my slideshow for a second so uh what I'm going to do is just kind of show an example of infrastructure as code used to provision kubernetes clusters and how I can integrate Cloud native operations into that so when a cluster spins up it automatically phones home into my instance registers itself and starts discovering and populating the cmdb completely hands-off so the way I did that was I created a a git repository that just contains all the kind of all the pieces that I need and I'll kind of Step through those in a little more detail after we're done with the slides but this does the spin up process takes 15 or 20 minutes so I just want to kind of kick that off so it can run in the background and we can have a true kind of live demo so in order to do that I'll Open my just simple Json file I created that in this imaginary scenario stores a list of my active clusters and I will just say well I got a request to spin up a new cluster so I'll just add that cluster name to my master list of clusters and then I'll push that update to my foreign repository which in this case I stood up in AWS and their uh code commit Repository and then I connected to that repository a code build project and um so I I put this together over uh about four to six hours so it's definitely very kind of proof of concept I do plan on putting it up a little bit putting out a putting out a community article on it so that people can actually kind of take the code and use it as a starting point or at least an example in the meantime I'll just kick off this build and then I'll get back to the uh I'll get back to the slideshow okay so the the main reasons why I would recommend using Cloud native Ops ease of setup is definitely top of mind yeah I've gone through prior to Cloud native Ops coming out I went through uh a lot of discovering a lot of kubernetes clusters and it was always you know enough moving parts that if it did become a little bit tedious after the last you know the first 10 or 20 clusters between figuring the um the bearer token and then creating a one schedule using the kubernetes pattern to discover the whole cluster and then a second schedule to run every five minutes to pull the events in um it was definitely if you not something I would describe as scalable so the fact that you don't need any credentials everything is deployed into the cluster using standard you know kubernetes delivery mechanisms makes this a lot more of a seamless experience also it involves fewer teams um in a real you know in a real world organization generally you would have the servicenow platform owner who would have to for example obtain the bearer the bearer token from the kubernetes team and then put it in the put it in the credential table set up the cluster um Discovery schedule the kubernetes team would have had to have created the service account which has the permissions to discover all the kubernetes components then generate the token pass that token along and then to test it out you'd have you know the servicenow side would kick off the schedule and the kubernetes team would have to kind of be available to work out any permissions issues whereas with Cloud native Ops the kubernetes team has a payload they deploy it to their cluster and it automatically registers to the servicenow instance and starts functioning um if there is a need for diagnosis again the kubernetes team has access to those mid-server logs those agent logs from within standard kubernetes interfaces that they can use to at least you know start the debugging process for figuring out what might be going going on that's preventing it from working Dynamic cmdb updates as soon as a pod goes away within in my testing within about a minute or so I'm seeing that pod marked absent in my cmdb if it was replaced by a new pod like a service you know a pod dies and by virtue of it being in a staple set or a deployment the a new pod comes up in his place I would see that new pod show up in in the cmdb as well with um you know with full dependencies on the or relationships within the cmdb it would show me the container or containers that were running inside it if it was part of a stateful set or deployment it would show that relationship right away so not just a little kind of a stub placeholder CI but a functional CI that you can use right away to see okay what's what service is this a part of what else is it depending on I did a similar test with labels earlier today right labels that translate into tag-based service Maps right that was you know a minute or two tops between labeling the pods and the services and seeing it pop up as tags within the cmdb so it's very quick very fast nice um events and metrics the um the turnkey so there's two components that come with CNO that are deployed inside the cluster there's a mid server and then there's an agent client collector that agent client collector is running some purpose-built checks which come out of the box which use the you know basically the ability of that pod from within the cluster to query the cluster API for ongoing events such as what I saw a lot of were scheduling events so when a pod would try and spin up if there was an issue with available CPU for example such that that pod couldn't be scheduled that would result in an event that would show up in my instance and that's that's not something CNO is the only out of the box solution that would give you something like that I mean you could put together of course the platform is a is a a big old Lego set so you could certainly enable that yourself um with customization but in terms of doing that out of the box uh Cloud native Ops is the way to do that and it also uses you know uses that same agent client collector functionality to also populate some basic metrics like available CPU um some Network traffic stuff which are available from the um the metric intelligence you know um workspace to you know you can construct graphs and and basically query those metrics the same way you would query metrics from any other piece of compute that's Gathering um that's Gathering that kind of data here's an overview of the way Cloud native Ops is put together uh the cloud native Ops components are collected in a stateful set there's some kind of supporting pieces that get put out there in the form of a service account and a token that that service that corresponds with that service account which is then mounted as a volume excuse me within the within the cloud native Ops pod so the current model just deploys uh a staple set with one pod two containers that runs on just runs on one node um if anybody tried out the pre-release of cloud native Ops um that had kind of played with getting a little deeper into the cluster by virtue of running uh an agent client collector as a Daemon set across which meant you had a copy of agent client collector running on every single node and then it was running in uh an elevated privilege mode so it could use TCP dump to pull down at live network connectivity there was feedback that that was a little too permissive for uh but a lot of enterprise um container security guidelines so that was kind of set aside for this this GA version there's still investigation going into how um how we could kind of still provide some level of that kind of Deep network connectivity information so more to come on that in the meantime in addition to what gets deployed we do support within Cloud native Ops the same recognition if an istio service mesh is in place that will allow the standard kubernetes pattern via the istio extension to connect relationships between Individual kubernetes Services and the way that shows up is uh uh connects to Connected connects to connected to by a relationship between the two kubernetes services that are talking to each other over that service mesh so we do still have that available within Cloud native Ops um Mike I think you looked at kind of the istio project and it didn't look like it was like can hugely adopted so it's unclear as to how how many shops actually make use of that today but the support for it is is there yeah and I expect to see that increase um as more folks you know continue down this kubernetes journey service meshes have a lot to Value to offer so I think it's good for us to start with that support now as that continues to grow I think it's going to be a more valuable thing for us so this is just a list of kind of watch items that Mike and I have compiled as we've been working you know Hands-On with Cloud native Ops you know just uh kind of call them Errata or just um kind of FAQ type items the first thing is if you're provisioning a cluster within uh eks so they Amazon made a change um a couple versions back maybe where um if you wanted to provision persistent volumes you they they introduced the requirement started as optional now it's basically it's required that you add their EBS cni add-on in order to provision persistent volumes and cloudnative Ops makes use of a persistent volume for the mid server to store some State information like the mid server identifier and and um the things that and like the the key store I believe is in there so that the mid server can you know survive past uh you know being moved to another anything that would kill the Pod and cause it to reinstantiate like moving it to a different node or if for some reason the midservice hung or whatever um and you need it to to kill the pod so in order to provision that volume you do need the the EBS cni add-on on your eks cluster and that's got its own uh I include some links later on in this deck with the the docs from AWS on how to set that up because it does require an additional role be provisioned to the cluster so that it's got the permissions to request those EBS volumes when they're needed and so there's an extent just a really quick thing to point out like if you if you run into this the way that it like shows itself is that the um the CNO pods just show pending like forever they'll never actually finish provisioning because it can't provision the back end um persistent volume that is necessary for the pods to actually spin up so it's it just sits there waiting and waiting and waiting for that persistent volume to show up for it to finish bringing up the containers and and it'll just sit in pending forever so um I had I actually ran into this problem will and I chatted about it I installed the evscni add-on and within moments they came up and everything was happy so it's not so bad just you should know and be aware of what that looks like yeah the the message and the the behavior when it's not in place is is definitely not intuitive you just get a pending and uh I think somewhere in the cloud watch logs you could probably get more information but it doesn't kind of jump out and and tell you so there the second the second thing I ran into is the the kubernetes pattern includes an extension called get kubernetes yaml files and what that does is it just pulls out yaml versions of things like uh deployments stateful set Damon Set uh Services I think might be a couple others and it tracks it puts them in the cmdb as tracked configuration files um so that extension is currently not compatible with Cloud native Ops what happens is if that's enabled on a cluster that's being discovered be it via Cloud native Ops it fails because it doesn't have the hooks in it to take advantage of that in cluster mid-server it's it's kind of it's trying to you basically follow the old kind of um agentless horizontal Discovery approach of looking for credential Alias and then pulling in a bearer token from the credential table so it just does not work at all via Cloud native Ops um I do have a case open on this just to get some clarity around it it does sound like at Large Scale this extension as currently written is problematic because it pulls on a large scale it pulls loads and loads of yaml and that was proving problematic for customers with large clusters so it's unclear whether that extension will continue in its current form or need to get refactored in order to better deal with you know the growing scale that um that people are are running these clusters at and that's something that you guys can help us with feedback on is is you know the the yaml file config tracking is that actually something that's of you know huge value to you guys is that important to you um you know if you give us that feedback and let us know then you know we can talk to the team and say hey maybe we should you know continue to work on this to try and refine it well or if nobody really cares then okay we'll just you know let that piece of functionality go and go focus on something that's more important for everybody so let us know please yeah absolutely and that's really part of the impetus behind trying to get this community going is to get that feedback loop engaged and constantly you know find out what's important what's less important so that we can feed that back to our product teams and and guide things in the right direction for all you folks mixed mode Discovery and so by that I mean a mixture of clusters that are being discovered via the old method with uh you know credentials in the credential table and running um a standard horizontal pattern driven discovery what I found was when I started with some clusters discovering them kind of the old school way and then I added Cloud native Ops there was uh an addition to the pattern that started failing for those old school discoveries so just kind of something to be aware of um I haven't had a chance to kind of run it down any further yet as far as um what's you know what's actually going on there but it was just something I noticed and and wanted to call out just to hopefully avoid some frustration on the part of other folks who might start you know kind of playing with CNO and also have you know traditional Discovery schedules out there that are that are running so we already mentioned that the um discovering connections between Services via istio is there um it's by not officially supported what I mean is by default when you set up a cluster with Cloud native Ops it does not enable that functionality in order for that functionality to work you've got to have istio you've also got to have Prometheus in place pulling um the istio connection data into it and you have to populate the Prometheus URL as uh within the kubernetes discovery schedule and cloudnative Ops doesn't do that by default so if you populate the Prometheus URL in that cloud native Ops schedule after it's created then those istio relationships will get brought in um but it is there's that is like a post step that you'd have to do and that could be automated any number of ways if you do you do happen to be running istio and you've got Prometheus in your cluster already you can kind of enrich your data further by doing that and then the last thing that we ran into is um Cloud native Ops provides a one-liner uninstall command which is available from the cloud native Ops kind of home deployment page within your instance we didn't notice a few um they're not break they're not issues that break anything but they the the uninstall leaves a couple bits laying around that you then have to clean up separately um so like the way I've been deleting things as I kind of cleaned up and then reinstalled Etc was I would just you know the CNO stuff gets installed in its own namespace it's user configurable I would just delete that whole namespace from the cluster and then the one piece that's left outside the the namespace is the service account and the cluster role that that service account is um is granted so those occur outside of the namespace at the cluster level so those would have to be cleaned up by hand if they uh if the uninstall leaves them leaves them lying around okay so I have one slide left but that's just talking about a change we noticed that kubernetes 1.24 so in the interest of continuity Mike you want to kind of do some of your Hands-On CNO stuff now yeah let's go take a look at it so um let me make sure I share the right desktop here okay so let me show you guys oh I'm getting ahead of myself there um let me show you guys so this is the kubernetes dashboard that gets populated I'm gonna make all of that a little bit bigger just because big screen okay so um first things first so when you install a net new um service for cloud native operations it actually adds this new section here called Cloud native operations into your filter Navigator and generally the first thing you're going to do is you're going to go and hit this deploy to Cluster link right so if you haven't already created an API key for your mid servers to enable the ACC agents connect up to your mid servers then it'll say hey we can't do that because you need to go create an API key and it gives you a link here to go and do that creation and then come back here it's actually quite nice so since I already have one I could pick which one I'm going to use for this deployment and I only have one in this particular instance so I've got that configured and then here are all the nice kind of one-liners right there's the you know there's more standard kubernetes this works on all of the magic excuse me managed kubernetes clusters as well as the vanilla Kates cluster and then there's the open shift cluster deployment lines to be blunt I haven't tested open shifts because I I don't run one but um the the deployment is actually really pretty straightforward you go to a a terminal that has Cube CTL access into the cluster in which you're going to deploy CNO you paste it there and what that does is that says hey here's the API key don't worry I know I'm showing all this this API key is going to exist for all about five minutes after this call ends so don't don't feel like you can You're Gonna Go hack all my ACC agents in a minute but um what this does is this actually pulls down the deployment script which it then runs the deployment script based on your inputs and it goes out it said it asks you hey what instance do you want it already knows what instance it came from based on the curl so it says by default you can use this instance or you can change it to another one um I and in this case I'm being lazy and I'm just using the admin user in order to do the install but you can certainly make a specific um user specifically for doing this deployment if you would like um for some reason is having an issue of copying my password um but once you enter the username and password let me see if this will work here correctly um one password of course is always the um the Workhorse and never fails me and now all of a sudden it's gonna fail me alive um let's see if it'll work okay so it's going to validate that in a minute we'll see if that works it's going to ask you if you're going to use Mutual TLS in order to do that we didn't set up Mutual TLS for this that is great if you can do that because then you can just provide certificate based off and it means you don't have to validate anything it just comes up with the certificate and and authenticates against the servicenow instance and it's all ready to go um excuse me I won't actually go ahead and kick this off and and for two reasons one it's actually going to download a bunch of stuff um and two uh I've got my instance set to Tokyo and by default right now this is for San Diego and so the upgrade process itself takes a little bit and slows things down so I don't want to make you guys sit through and watch paint dry and do that um but so then it says what namespace do you want to use and I always just take the default of servicenow to keep everything in the appropriate namespace and it it's actually gone through and deployed everything or attempted to um and in this case because I already have everything deployed in this cluster um then uh it just said unchanged for almost everything that was there once you do that for a brand new deployment and it's the very first time um we'll go back here and what it'll what it does is it now that script reaches back into the servicenow instance and actually kicks off what's called a deployment status tracker which is really really cool um it kicked off a fresh one that's kind of interesting um uh but I can show you one that's actually gone through and completed and it kind of walks through these whole steps one by one by one right so it actually goes through and deploys it um it registers the mid server meaning that it comes up it actually sees the mid server registered to the servicenow instance and then it waits for the the mid preparation step which at that point that is if if you're not using Mutual TLS then you have to go into the mid servers list and validate that mid-server right once the mid server is validated and there's actually a link in here in order to do that if you need to then it goes and starts the um the mid web server extension to enable the agent client collectors to connect to that and that takes a minute for that to get up and started and then the agents have to connect to that and then what it does is it goes through and it creates a discovery schedule specifically for that cluster and then it pushes the patterns down to the mid and it triggers the discovery schedule for the first time let me show you what that Discovery schedule itself looks like so I'm going to go here and we go to Discovery schedules this is one of my nitpicks that I'll be talking to the team about is they all come with this name kubernetes discover from mid and cluster I'm going to ask that they change that to a more descriptive name because if you're doing this across a thousand clusters for a large scale customer that's going to get a little um repetitive so and you'll have to kind of dig through to figure out which one's discovering which cluster and so once you click into it though um you you actually kind of dive in and you can see in the pattern what it does is it uses the kubernetes service address from inside the cluster to do the discovery and it does that because you know as you know kubernetes has its own it's sort of internal DNS inside of the cluster and so everything has its own you know nice neat easily reachable names for all of those services within the cluster same thing with the Prometheus connectivity for doing the istio ingestion of alerts and metrics um it uses those nice pretty internal names um in order to do that so great question Richard I'll answer this live here while we're waiting for my Discovery schedule to load for some weird reason um does this kubernetes discovery happen only as often as specified by the discovery schedule or is it happening constantly from the agent running inside the docker image so how this works is there is a it creates a periodic schedule that runs on a daily basis to do an entire whole cluster discovery and then um on a on an ongoing basis it's doing real-time updates constantly from the agent running inside of the cluster in order to ensure that we're getting new real-time updates from that so it's um it's going to be very very close to real time for all of those major changes but it's not doing a full cluster Discovery every couple of minutes it's doing um and I apologize it looks like that thing is just giving me a hard time um that it's just doing like incremental updates from events as they change within the cluster uh and and then doing a full cluster Discovery once a day so um here's the execution pattern um once we get into that you'll see it's using um here's the cluster URL right it's literally just the cluster URL right um it's not using any probably available IPS none of the traffic between the mid server and the kubernetes cluster are actually leaving the cluster because it's going directly up to the API server from inside the cluster and then this is what will was talking about it doesn't by default because it doesn't know the address for your Prometheus interface within the cluster in order to be able to ingest metrics and in istio data from that so you'll have to go in and manually set that Prometheus URL in order to be able to get that data in so once you've got that set um and it's activated it goes through and it does a discovery um and I found two interesting things about it right now um one it actually creates a out of the box that creates a service in the operator workspace specifically for the cluster itself right so when you go in and look at the service you know here's your here's your cluster ID it comes in as a mapped application service it gives you all of the CIS that are all related to that and then we can actually go and look at the service map and it actually builds a a solid service map around here's the name spaces here's all the other components that are within that cluster um and that's all and that's all mapped out automatically this isn't dependency viewed right this is belt mapped by Cloud native Ops right so here's the here's the nodes pool right um here's the various namespaces that are in that and then you know you can get get pretty deep and dive deep on it one of the things I did was as usual I went in and installed the Google microservices demo right the boutique app and got that in and you know here's all of its related CIS this is all um you know ingested and built that way and then I built it out as a tag based service map I am using istio in this cluster um and so I've got izio sidecars being injected into all these Paws in order to do the service mesh capabilities right so um and this is you know this is a standard tag based service map but it's all all of this data is all being discovered real time via Cloud native Ops and I can go and we're running low on time so I won't do it but I could go and apply a whole new tag set to all of this and it would all show up immediately in the new tag set because it's very very quick um okay I'm gonna go ahead and hand back to you will it's cool um so you can go back to your pipeline so just um an observation about the way the dynamic updates seem to work there's a it it's all done via an ACC check so the agent client collector that's running inside your cluster is running a specific check in the form of a a script which is gathering it's it's polling for events and then based on that it's constructing a payload which it then passes back to your instance and Cloud native Ops defined a new check type um this to specifically address those payloads and the back end script which handles that payload that um that check type is what's actually kind of preparing the ire payload to reflect oh this pod got deleted this pod got added it does seem to be kind of focused on pods and the things that stand pods up and and tear them down um in my testing it didn't look like if you do if you do like a new node if you take a note away or if you add a node I didn't see that reflected in real time so that level of change would probably require kicking off the full Discovery schedule in order for updated node counts to exist uh I'd also like to just say that Mike and I did not coordinate our hoodies today um completely uh serendipitous yeah I noticed that too yeah okay um so just to dig a little further into my automation use case so what I put together was just a quick set of some scripts and some kubernetes manifests which would stand up a kubernetes cluster and then after the cluster came online deployed Cloud native Ops to it by default out of the box cloud native Ops is oriented towards kind of an interactive install as Mike kind of walked you through what I did was I just took that one liner which part of what that one liner does is it pulls down an install script and a manifest and the install script pulls down a manifest it modifies the Manifest customizing it with things like your mid server password um you know your mid-user password your instance name if you're doing TLS it plugs in the TLs information uh so I just kind of bypass the interactive elements of that and just put a quick um CI CD wrapper around that so just to kind of run you through the base of the main files that I set up um this file here just was required by eks to make sure that I had access to my cluster after I spun it up because by default if you spin up a cluster using a devops user or a standard kind of service account that your cicd pipeline is using that user is going to have access to the cluster but not necessarily you and we've got you know Federated um you know IDs that we use to access AWS so I just had to create uh a quick manifest that would then kind of impart those permissions to the service roles that I'm creating as part of this script which is actually what uses eks cuddle in my case since I'm working this example is all in eks but the same principles apply regardless of what cloud provider you're using the specifics will vary but it's it's all perfectly feasible within any of the standard Cloud providers so this is just a quick wrapper that I created to use eks cuddle to create a cluster um to apply some standard logging to apply our standard IP white list and then I just created a a python script which would kind of take the combination of this clusters Json file and this utility script and this kubernetes manifest and just kind of pull it all together so it starts by and so the reason I did it this way was so that I wouldn't have to embed any credentials so I've got my my kind of sensitive bits my credentials stored in AWS Secrets manager and then I just make some API calls to pull those secrets from Secrets manager within my CI CD Pipeline and then I populate those into environment variables which is all fine because all this is running inside an ephemeral container there's no server that somebody can break into and you know pull environment variables Etc and then um like I said I'm gonna I'm you know don't feel like you have to capture this real time or anything like that and I I'm gonna kind of make this a little more generic and then I'll publish it as a a community article and the link to that article will be included in our follow-up email for the session but just kind of continuing the overview is I I just have this script run a few external commands I have it stand up the cluster using that eks cluster script passing in the name for the new cluster and then I do kind of the the rest of the stuff that I found through trial and error that needed to be done to provide like me the access to get into my cluster once it's there which you don't get by default the interest of lease privilege um this has some code to make the EBS stuff work add the EBS add-on and then at the end I just take my kind of uh slightly modified and customized version of that interactive CNO install script and I invoke that and it takes those environment variables that I populated out of AWS uh secret manager to fill in the sensitive bits and then the result you know for my for my reading pleasure I can kind of scroll down through the the full build process so if if I had had an issue I would see you know I could see any errors not sure why I puts all those blank lines in there um but um there you go the mystery of AWS and so long story short I end up with a new cluster which is provisioned via a handful of cloud formation Stacks one for the the cluster control plane one for the node group uh here and then a couple add-on rolls one for the uh EVS role and one standard role for the AWS node itself so it could communicate back to AWS and then what that looks like on my servicenow instance is uh again my as Mike mentioned we didn't go through uh the mutual TLS setup in either of our cases because that is that's a little rigorous uh especially when you don't have access to uh unofficial certificate Authority so you do get that one kind of manual step if you don't use Mutual TLS you do have to do the click to validate the mid server but so now the process will continue and I'll end up with a kubernetes cluster that my instance now has full visibility into and is aware of Without Really without the servicenow platform admin having to have done anything if you're if you're set up for Mutual TLS if you're not again they they do have to do the one click to validate the mid server but um for any kind of Enterprise a standard pki and CA set up the mutual TLS stuff is is not is not bad and imminently um can you know imminently Autumn automatable if if that's a word um and honestly if you're okay with the all your mid servers signing in as the same mid-server user you can generate one certificate assign it to that user and then have that in a Secret store like a native us Secrets or something like that that the containers can then pull from and have access to that one certificate so it's definitely got uh automation capabilities available one other thing that's just worth kind of touching on is this is an example dependency map that shows the results of when you do have your if you are using an istio service mesh the fact that you can then see the dependencies between different services so we'll just kind of pull this out so it's a little more visible so you know for example we've got the checkout service within my example app and you've got all of these connects to connected by links back to the different microservices which make up my application which is really useful you know from a change management perspective from an incident perspective to kind of um pull together a really accurate picture of impact for either of those two kind of scenarios and then um let's see I think that was about it Mike did you have any other CNO things before I go into just a little uh note about kubernetes 1.24 no we should we should cover one two four okay so we just noticed kind of a little bit of a a change to the behavior when we started working with uh with clusters based on 1.24 and this pertains not to CNO if you're using CNO this is not an issue but a lot of people aren't using CNO they're using standard patterns and so we just wanted to bring this up and I actually wrote a little community community article about it that includes uh you know a fix or a workaround whatever you want to call it and the link to that is in the deck here that we'll be sending out so prior to 124 the servicenow instructions um you know basically said create a service account or you can use the default service account but if you created a service account for Discovery you would automatically get a secret that you could pull out of your cluster and put in as the bearer token credential for the cluster within your credential table and we noticed uh you know of course while working with a customer that um they were creating a service account but there was no secret and then you know we looked through release notes Etc and found that that was a deprecated functionality and you no longer get a secret kind of free of charge you've got to actually do an extra step to create a long-lived secret they basically they're going for service accounts they're going to a token based approach where it's almost like uber it's almost like um Kerberos where you kind of just periodically get a fresh token and that works great from within the cluster but if you're looking for a bearer token used from outside the cluster to discover it that doesn't work so great so there's you know they still support generating a Long Live Secrets just requires an extra step and um as I mentioned there's an example manifest that you can use to generate that in the article that I that I put out on the community site so just really to kind of spread the word and let folks know they run into challenges trying to use the standard pattern-based Discovery it's kind of expected Behavior once you get up to 1.24 or higher okay um so we're going to go into q a just to kind of call out I did add some links pertaining to stuff that I had talked about um and our next session is going to be on the 23rd of February topic is still TBD as always we welcome topics suggestions from our attendees so feel free to reach out via email or you know however whatever Channel works best for you to suggest future topics uh also just a quick plug we are still working to try and get a live beers with Engineers session going for knowledge 23 in May um more to come on that as we hopefully make forward progress we definitely want to drink beers with you guys in person um and and provide beers and other beverages of your choice so that's definitely part of what we're working on with regard to the live uh the live session