all right welcome everybody we're officially recording um this is session 15 of beers with Engineers um with your hosts will Hallam Mike Gallagher um super excited to have everybody here with us today we are going to be diving into some custom containerized builds for midservers and agent client collectors so we'll go through kind of the the usual intro for those of us um who haven't joined in on this um with us yet um you know we'll go forward well if you want to pop up the agenda we usually have a pretty um pretty light agenda and also we always have to start with the Safe Harbor slide right this is an important thing um there are things that we talk about here that are either not released yet might get released might be on the road map might never get released so don't make any purchasing decisions around stocks or any of that stuff uh based on the conversations that we have here today so now on to the agenda right it's always kind of the same why are we here who are we and then we just get right into the really deep nerdy stuff around like you know the technology that we're working on and what we're what we're diving into today so um so we'll talk about why are we here right the goal of beers with Engineers you know at the outset really is to is to bring together this wonderful community of customers to talk about you know what people are doing on the cloud native space and how they're utilizing the servicenow platform in order to kind of manage that and and migrate along that cloud native journey and to manage that technology portfolio and um I'll just really quickly want to call out the the community aspect of this is is really a big deal and and at knowledge last month we actually got a chance to do this live um with a live audience at knowledge and that was just amazing uh to meet everybody here that we have been kind of hanging out and chatting with for you know a year now right um so to meet everybody in person and Shake Hands was really great and so thanks to everybody who came out for that and we're super looking forward to hopefully being able to do it again next year too so thanks for that so who are we right uh so my name is Mike Gallagher um I actually run the Enterprise applications platforms team for drw Holdings um a trading firm out in Chicago um and I have been doing technology for a really long time um longer than you know some people have been alive it's a little scary uh but um I uh I'm highly focused in on kubernetes I.T operations management um you know cmdb population IIT Asset Management I'm very big on you know utilizing technology the right way to solve business problems and and focusing in on the business value but also you know kind of the shiny cool technology too right um and of course you know I'm a big nerd on multiple levels I love board games Jiu Jitsu all that fun stuff um and frankly I have a crazy amazing family that I love to spend lots of time with so that's me and that Michelle will thanks Mike uh will you be enjoying a beverage today I will I almost forgot today I am having a an Avery little rascal um which is a a session Belgian style white ale so Avery's a local uh Brewing Company here in Colorado I normally drink Stouts and they make amazing Stouts but I figured I'd go with something light because it's pretty summery out here right now so so here we go here's to the The Little Rascals now will it's up to you hey everybody for those who met up already will Hallam I am an itom architect with servicenow then technology is super long time uh last decade or so focusing on a lot of cloud native Technologies public Cloud uh kubernetes and automation within that space and uh in my spare time I also enjoy hanging with my awesome family playing pickup hockey and video games and today I'll be enjoying a juice bomb IPA from Sloop Brewing okay so test my ability to multitask and pour and and speak to slides at the same time so this slide just captures a few you know a few reasons that we've experienced or are hearing from customers for customizing creating custom um container images for your mid servers and then um you know we've had that capability for a while and then a slightly newer capability the the cloud native operations um plug-in which uses an embedded mid server and agent client collector in kubernetes clusters so so one of the things that we can run into a lot is uh as container technology is maturing companies are you know implementing governance around that and placing requirements on container images around things like you know vulnerability scans or just um you know practices and minimization or they've standardized on a given Linux distribution and setting that as kind of the Baseline all containers all Linux containers need to be based on that distribution similar with jres JRE usage is you know another area that companies tend to standardize on and so by creating a custom image it gives you the ability to twiddle all those dials that you want and come up with the the most compliant container image to support your containerized mids or your uh your Cloud native operations payloads he's here debugging this is something that I've run into firsthand is um you know that's that's kind of one of the challenges in general in the containerized world is it's definitely not as easy to just jump into a container and see what's going on compared with if you just got a mid running on a Linux box or Windows box you can log in you can look at log files you can add you know print statements Echo statements to Powershell scripts or or shell scripts it's fairly accessible to somebody who's you know an I.T practitioner and trying to figure out what's going on with the mid server or trying to assist the platform owner um once you're kind of going over to a container image it's more of a black box you just throw it down there you run it you give it you know you Pat you're kind of at uh the mercy of whatever hooks the container image author has given you so there are some of those out of the box and we'll cover a couple of those the ability to tweak the mid-rapper settings that's provided with the standard off the off-the-shelf recipes but building your own image gives you way more ability to kind of inject those debug statements those Echo statements Etc if you're trying to debug or debug a mid-server behavior or perhaps just kind of um you know develop an in-house capability that requires more transparency into what's going on inside the mid flexibility um again just giving you the ability to access to all kind of all the dials in terms of what goes into this container image and what it looks like when it gets deployed to your various container run times plugging into existing CI CD tools again as this you know as containers and kubernetes matures out there in the field companies are developing standards around not just what goes into the images but how the images are assembled they're using standard CI CD tools whether it's Jenkins Ado Amazon code build whatever by building your own mid image you can just plug right into that existing um you know that existing capability within within your company and a lot of times that removes a lot of friction when it comes to hey we want to run this mid server image inside all our kubernetes clusters for example or deploy this in an elastic fashion and five five kind of ties back to the things you know company standards you know and um it's really just another aspect that companies are internalizing they're bringing in artifactory and they're using that to host all of the images that they run uh you know there's there's there was certainly a lot of unrest due to various you know supply chain related breaches like the solarwinds issue and um you know coming from the customer side all of our kind of payloads like that whether it was container images install packages we grabbed none of that directly off the internet it was all coming in to an internal repository like artifactory and then all of our live workloads were pulling from there and so by building your own image and just incorporating that into standard pipelines that your you or your customers have it just kind of slots it right into an existing practice you don't have to reinvent the wheel you don't have to you know have a bunch of kind of manual processes put together to take an image and then you know push it into an internal repository you can just push it right to it any other thoughts on what you're seeing driving um you know you you as a customer as the owner you know when you when you're going to be going to your kubernetes team saying hey yeah we want to run this containerized mid what what are they asking you what are you anticipating they're going to ask you for in order to kind of say yep that's cool yeah so for us you know um everything has to come out of our internal container repositories right so you know nothing nothing external so if it needs to run on an internal kubernetes cluster right it has to be pulled down built and published into our repos in order to make it Deployable so just that simple fact means I have to I have to do the builds right but then as we start thinking more about like um did we talk about the cloud native operations component right because that all needs if we're going to do that now we are going to kind of emphasize that for sure because that that was something that kind of was one way and then it went the other way yeah so that's you know there's and then there's there's other components like you know maybe um you know making it um available to you know certain clusters right maybe we have restrictions on um using um kind of like Secrets off for being able to pull down you know from your own um from your own distro excuse me your own repo right those those kinds of things are things that I'm seeing customers all over the place needing to do right um so that all makes sense the other thing is you might want to I it it this is difficult right it's not an easy process but you might want to use a different base um image based OS image for doing a container build or or a CN or excuse me a mid-server build or an ACC build so both of those things could potentially require that you might want to do your own um your own build out yep for sure so this next slide highlights just some of the uh some of the things that kind of jump out when you start looking at the standard recipe for a mid-image um and as well as just kind of overlaying that on feedback that we've gotten from customers uh customizing the Linux distribution um standardizing on a given flavor of Linux that that's nothing new um and we're seeing that kind of propagate over to The Container side if you're going into a red hat shop and you say ah here's our image you can run and it's based on Ubuntu um you know that that's probably going to be a bit of a source of friction at some point when it hits you know the various teams that are expected to support that image right so by building your own you can basically I mean you can get pretty deep into it I actually um when I went through this exercise I initially tried um so for those of you out there who work in Linux you may be familiar with alpine Linux which is a very lightweight distribution uh it's like everything in it is statically linked by default so it's very kind of um it was created with the idea of using it to create Linux containers with the idea that it minimized the tax surface it's a very minimized distribution when it their base image they provide has almost no packages and then you can build stuff out on that but the idea was it would give you a very small surface area to start with um unfortunately that I I kind of got down a rabbit hole with alpine because our because the mid itself there was one shared library that the mid was trying to access that was not statically linked into that Alpine kernel so um in the interest of kind of you know I'm kind of doing this more proof of concept than uh than a direct you know customer asks so I kind of just uh realigned a little bit and I went with Alma Linux when I did my example because that's kind of similarly it's it's better than the default Centos 7 because Cent seven is about to go completely end of life like in about six months or so so that you know is just a lot more venerable of a distribution and so I really just wanted to kind of see what was involved so if you're using kind of a red hat variant it's pretty straightforward if you want to switch to like Ubuntu then you have to rewrite more of that recipe because it uses a different package manager and probably has a few different package dependencies that may not be incorporated into the default recipe uh customize the windows runtime if you you know Windows running on containers is still a bit bumpy but if you're talking mid-server and you want to do windows discovery or Windows automation anything that needs Powershell you gotta have a Windows mid uh still for now so um there's actually a requirement that the the windows image that your container is based on be the same version of Windows server that your node is running that you're that you're um you know if you're running kubernetes that your kubernetes node is running so you know if you've got a 2019 node and the image is based on 2022 or vice versa it's going to say not supported and not let you run that image so customizing the windows runtime might be a more attractive option versus trying to get some non-standard kubernetes node spun up just to run that Windows workload I have uh customize the jdk that's another you know just um see see to the pants I I didn't see major issues with you know our jdk I was a little thrown off because when you look at the recipes it pulls down to jdk8 base container at the beginning so I saw that I was like well that doesn't seem like a great idea but then I realized that's really just to run the downloader to download the payload if you're using a if you're not using a local mid install payload so it can check the md5 signature and verify that it's a valid um mid-server download so the the jdk 11 that we use is is from a you know kind of a currency perspective it seems to be not bad but again there's corporate standards a lot of times saying we want to use this jdk and it's very easy to customize by building your own image um we have a nice little hook in place where you can use environment variables just out of the box if you download the the uh the recipe to adjust your Java wrapper settings so that kind of actually is complementary to number three because in order to customize jdk you have to tweak a wrapper setting and that is like super easy to do just to find an environment variable for whatever wrapper setting or settings you want to adjust so if you want to you know change your your stack size for example which is a big one if you're pulling in a lot of events or doing HLA on your mid you sometimes you need to increase your stack size and then another cool thing we provide is with with a couple caveats is um the recipe includes a mid-secrets properties file which you can use to kind of set up configure the secrets that you want to push into your not so much your image but the secrets you want to you want the image to be aware of so it can pull credentials in when it actually gets deployed and I know Mike you ran into a little bit of a uh uh incompatibility because the secrets files in one format and it reads it fine if you are deploying it raw onto directly to a Docker you know like Docker swarm but if you're using kubernetes there's a slight difference in secret format so you have to kind of I don't I looked at our docs really quick and I didn't see it didn't look like they really updated that with any kind of notice it just says set it up as a kubernetes secret and then add the secret to your manifest and it I think your experience was it was a little more involved than that to get it to pull those secrets in if you're doing it in Kates versus raw Docker or swarm it has to do with the way that the secret is mounted into the mid-server image right so with pure Docker right it mounts it into the running container as a file and the key value pairs are just lines in that file but when you define secrets in kubernetes by default it's actually it's mounted in as a folder and then there's a file and there's multiple files in that folder based on the values folder directory whatever you want to call it um based on the values that are set in no secret so it um my experience in the past when I try to get this to work correctly was that it required some some heavy tweaking of that script to instead of because it's looking to open a file and then Loop through the lines in the file um and that did not work and I had to modify it to actually read the directory and read each of the files in the directory and and modify the variables based on the files in that directory so it it took some tweaking but you can make it work yeah yeah it's a good the fact that they kind of recognize the need to to not throw everything into everything into a clear text or an unmasked format was good um and hopefully there'll be you know some kind of an Errata in the documentation or or whatever down the road but in the meantime it's just something to be aware of if you want to use that mid-secret properties file and you're doing it on kubernetes you may have to tweak the um there's actually a script uh that I'll kind of touch on when we get into the live live demo stuff called uh the init script where you'll just have to tweak that shell script code if you're going to be pulling it from a kubernetes secret based on the the difference in the way it mounts those um I just realized we were kind of remiss and just um reiterating the fact this is a very kind of informal conversational webinar series that we have here so anybody who's out there feel free to come off mute with any questions interject at any time we don't really stand on formality um and you know generally we've got over the scheduled time available to kind of go down whatever rabbit holes people want to so so don't be shy out there hey and to that point just really quickly I want to call out uh Varun had a comment in the Q a um around that they would prefer vendor provided Docker images that way it's vendor ownership to fix any security vulnerabilities right and I totally agree and understand with that that's definitely a stance that a lot of organizations take right um where you know my organization's kind of in the same boat however there are still times where we need to take that you know the base recipe right and then make a few tweaks to it but still right like the kind of the Baseline image is still really owned by the vendor and we you know go back and push on them so it there's definitely kind of The Best of Both Worlds right there that you need be cognizant of you know you can't just take the take the vendor provide an image all of the time sometimes we gotta tweak it but at the same time right like I hear you it's very nice to basically to use an old phrase like have one throat to go choke about like hey go fix this problem right so um I'll mark that as answer live and then and I always right feel free to jump in chime in with questions or comments uh in terms of customizing Cloud native OPS images so as we alluded to already that's actually required now we did start out with some publicly available images unfortunately um [Music] because of the way vulnerabilities are flagged like I said it's kind of a yes no and there's very little con ability to provide context as to what the actual impact is so the decision was made rather than you know cause a lot of concern because of those um because of the current practices in place we'll just kind of go back and align to what's been in place for a while with mid servers which is basically will provide um you know we'll provide a recipe and allow the customer to kind of determine where to go from there so one nice thing that they did include they were kind of forward thinking when they came out with the manifests that get pushed to the kubernetes servers it already includes a commented section that'll cover you know when we get into the Hands-On stuff where you can pull you know you can basically just there's a table in your instance you just modify the yaml and from then on when you invoke the kind of um one-liner to install Cloud native Ops it will pull from your private repository instead of trying to pull from a public Repository uh one thing that Mike and I noticed is that the off-the-shelf recipe when you do a when you do the docker build it has a command in there to copy everything out of an asset subdirectory um and at least in the current kind of shipping version that you get if you just install CNO from the store there's nothing in the asset directory and the docker copy command at least on the platforms I built with uh it doesn't like that it's like I don't find any files and it'll just throw your build out so you really just have to so the intention of the asset directory it's kind of a a generic construct where if there's supporting scripts like there are with the mid server or if you want to embed the actual um Mid or agent client collector um install payload in your as opposed to pulling it off the internet is you want to incorporate that in your build tree that would go into the asset directory but by by you know by default when you try to run the ACC recipe through it doesn't have anything in that asset subdirectory so I just touched an empty file in there and it allowed it to it allowed it to proceed yeah that's an important call out it's not on the mid-server build it's the Asian Supply collector one yeah yeah I was like there's plenty of there's a couple different scripts supporting scripts in the mid recipe that exist in the asset directory and for whatever reason none were required with agent client collector and so if you don't happen to be using uh a pre-downloaded local copy of the ACC install package then that asset directory is going to be empty um I think Mike you mentioned you had an issue as well with the agent client collector recipe using an old like an old Debian distro that like caused issues because the repos had been moved to the archive or something and yeah that's right I had to um I had to change um I had to change the man now I have to change like Debian 11 or something to get it no I didn't I don't think I changed the base what I did was um I changed the I added a little script in in the build out to add the right archives directories for the the package manager to be able all the packages off the archive because the that that version is so old now that all of those packages have been moved to the archive version as opposed to the the primary hosted version of that package repo yeah I mean you had tried it before me so when I tried it I changed my recipe to use um Davey and 11. actually I did Debian latest okay which didn't work because apparently they've deprecated some package the deep deep dpkg-sig package which our recipe uses to validate the signature of the ACC install payload yep so for some reason like Debian 12 doesn't have that package so I had to actually go to Debian 11 which is a pretty current distribution yeah but um yeah so that was that was a little bit something to watch out for um for folks who are looking to build an HCC image and and maybe want to use something newer than than Debian nine um and so just for posterity I just put the name of the table in the instance that I'll be showing in a couple minutes that contains those yamls uh that you have to tweak if you want to point um your CNO installs to your privately hosted image and so now Mike Mike ran into some interesting um Adventures trying to build Docker images on his Mac so he was going to kind of share that so hopefully folks can um not have to experience his pain so so I I have a new M1 Mac right and so by default if you do a Docker build it assumes that you're gonna build in the Linux arm um style platform right which if you do that your your image won't run on most kubernetes infrastructure because they're all on on like the AMD 64s type um architecture so um I had I banged my head against this a little bit and then I did some digging around and I found somebody out there had a Blog article around using build X and trying to um and trying to kind of build like a customized build environment to do this platform that unnecessarily over complicated things and so this process that I documented here and shared there's a Blog article listed in here it's dramatically more simple than that leave it to me to find a more difficult solution than necessary but this actually has worked for me quite well you just have to specify the dash dash platform um you know build ARG as part of the doctor build process and work like a champ got it built got it pushed up into the into the repo and then it's often run to the races now nice you mentioned buildax I I did notice that and I was using um Azure devops and AWS uh code build to build my images I noticed one or maybe both of them the the containers that they were spinning up to perform the build um we're kind of chastising me saying you know the docker build function is going away and you should really be using build X so at some point I don't know if that means it's going to get more complicated or whether build X will kind of support some of those same kind of pass-through parameters to specify platform but um I did notice that build X is kind of one of the future directions if not the future direction for for building uh for building images so so that's in terms of prepared slide content um what I wanted to do next was I was just going to kind of walk through a couple examples of container image builds that I've been through for the last pretty much six weeks since we got together live in knowledge um I'm going to start with a general comment man at least for somebody who's kind of a you know a part-time container Aficionado such as myself building container images is a real slog sometimes yeah um it's it just because you don't just have the ability you know like I talk like I talked about earlier just log into the server and see what the process is doing and look at the log files it's very the iterations are sometimes um a bit lengthy just to make a slight change and then put through the build and then launch the container and then try and pull the logs out through different mechanisms um especially given the overall direction to go towards very minimized containers so you're using a you know fortunately I guess it's a plus um about the the fact that our our both our mid and ACC containers generally run within a fairly full featured OS base image you can still um if you're running you know if you're running a local cluster or if you're testing with a local Docker runtime you can still generally just kind of go into your container kick off a bass shell and kind of poke around a little bit but one of the use cases I was doing was based on AWS fargate which is um basically you know it's containers as a service you don't have any access to the nodes you can't do a Docker you know connect to a fargate container to just kind of poke around so that in particular was it was fairly time consuming but it was kind of a cool um use case and so I figured I would kind of show that I'm actually um wrote it up in a a community article as well and I put the uh the files that I used in uh in a git repo that's linked in that it's linked at the end of the slide deck that we'll send out to all of our registrants and it's also linked in that Community article and um it wasn't an idea I came up with by myself it was uh somebody who could come across one of our videos and had a customer who was interested in having very kind of as as as throw away a mid-server as they could possibly have so they didn't even want to have to host a kubernetes node or a you know a cluster runtime node in their AWS account they wanted to use fargate and they actually you know they were able to kind of throw the basic image out there but the problem we ran into was every time it would go away and respawn it would show up as a A non-validated or uh either non-validated mid or admitted need of re-key and they could kind of mitigate that with mutual TLs but that was you know they kind of saw that it was kind of a Band-Aid so what they're really looking for was a container that could run in fargate have some kind of persistent storage so that you could relaunch it as many times as you want it would just pick up where it left off instead of getting this net new MID server every time so that was what I kind of set out to do and so just to kind of run through the the tweaks to the existing image that I had to do to make that happen um and I did some other I did some other stuff too um I think I might have adjusted the uh the base windows image just to kind of this whole thing was basically kind of proof of concept so I may have tweaked that just to kind of verify that that would work um and then actually okay so yeah so one of the things so the the approach that I came up with and I did actually do a little diagram of that just to kind of try and make sense of it for myself um so what I did was I I used a couple AWS capabilities uh fargate today if you're running Linux you can have persistent storage in a fargate container using their EFS which is basically their cloud-based NFS um and it's baked right into the definition of the of the E of the ECS task so like if I actually look at my Linux version of this and the Linux one was way easier uh I'm I'm kind of I have predominantly like a Unix Linux background um I I do you know I do lots of Windows stuff but I'll run into these weird Windows things and I'm just like man this is why I prefer Unix and and Linux so um so the way I configured my Linux based fargate container I just kind of bring up the user-friendly display to do the persistent storage all I had to do was go down here and say yeah I want to mount up an EFS volume and then I had to find an EFS volume with a EFS file system and I just said yep just mount it up and then here's the directory to stick it under and we do provide in our out of the box recipes uh a specific directory that you can mount persistence persistent storage to we provide that for both the windows and the Linux recipes if you mount up persistent storage to that directory it will the the recipe will copy files back and forth to try and preserve some of the state of that container when it gets killed and comes back what I found is the container the container image for a generic mid server as opposed to Cloud native Ops it didn't store the key store it wasn't copying the key store back and forth for whatever reason I would think that's probably a pretty easy fix I'm sure that's in the works at that point that'll make like the the validation and the key getting out of sync go away in the meantime it was easy enough to kind of tweak as I was doing my Custom Image but anyway long story short this capability here to take an EFS Mount and just automatically transparently make it available to your windows fargate container that doesn't exist it's it's not supported yet and I asked Amazon smarter said is there any way to do persistent storage a Windows container and they said not yet here's the link to click on to request that enhancement so I just kind of rolled my own and again and this it's heavily caveat in the community article this is no way ready for prime time it's a solid proof of concept I got my mid server to spin up and I can kill it and restart it and it doesn't immediately go invalidated or in need of re-key the windows mid also uses some kind of an encryption for the if you're using username password credentials for your mid account it does it encrypts that using some key based on the user of that OS image which means if you restart or kind of kill and restart the same container image and try and point it at you know a persistent mid-server file hierarchy it can't decrypt that password that was encrypted because it generates a new key every time you relaunch the container so um fortunately the out of the box container recipe provides a force option that basically says Hey repopulate the config.xml every time you launch and and that got me over that but it was just another kind of iteration in trying to kind of get this thing working so what I did was I just took Amazon FSX for Windows which is there um SMB share as a service tied it to an AWS managed ad namespace where I Define my mid-server user because that's kind of a hard requirement you have to tie if you want to create an FSX share you have to tie it to uh some type of AWS fronted ad whether it's like a Federated or fully managed within AWS that's that's up to you uh anything that was sensitive I stuck in AWS Secrets manager so there's nothing you know being passed in the clear as far as like the width the mid server account credentials to get to the instance um I also leverage that for building the image for my Docker Hub credentials and things like that and then I've got my container image repo which I actually um even though I was running this at AWS I used Azure devops to build the image because Azure surprise surprise has a much better story to tell with regard to spinning up Windows containers and so the windows containers you spin up an AWS code build they don't have Docker on them so you can't use them to build Docker images so you could build your own image that did have that support but that was like one extra step whereas I could just go over to Azure devops and it would just do it for me so those are kind of the all of those kind of components went into my fargate task which spins up one of my shiny new windows mid images and then um copies what it needs to to the persistent storage if the persistent storage already has what it needs then it just creates a Sim link to the uh to the persistent storage and that was uh a change I had to make to the docker file so by default and and rightly so our um our Windows mid server Docker file switches the the user by default Windows containers run as a user called container admin which is basically an admin user and following best practices by by default our container images switches that user to container user which is a non-admin user because normally there's no reason for a mid to run as an admin on the Windows box and so in keeping with that our recipe switches the user accordingly the problem I ran into is and this was one of those Linux versus Windows moments for me you have to be an administrative user in Windows to create a symbolic link and the whole basis for this exercise was creating symbolic link so that the agent directory instead of existing in the container image it would point out to an agent directory that existed on my on my network share so that that would be persistent past multiple runs of the agent and again um in the quick and dirty Department um that should definitely be skinny down so like the binaries and stuff still stay in the container image I don't know if Windows containers will work running like with the OS partition read only but I know that's like best practice wherever you can do that so you know in keeping with that a production implementation of something like this would want to have just the writable files the kind of the transient files that need to change sticking on that persistent storage with the static files residing in a read-only kind of os partition within the container image but anyway I basically so I threw out all that code that we do off you know off the shelf in here which is basically changing the ownership of the all the mid files to this container user I did play around for several hours with trying to use um the SEC edit utility to edit the local security policy in the container so that the container user could create a symbolic link I'm sure it could have gotten it eventually but after a couple hours I just said I'm just trying to kind of prove it could be done here so let's move on and I believe I I do call that out the community article that you know if somebody really wants to run containers containerized Windows mids and fargate they'll need to solve that problem before they can before they can go live um and then the other thing I had to tweak as far as Docker file is concerned like I mentioned it does like a it does an encryption of your mid password if you're not using mtls if you're using mid username password um it encrypts that the first time it starts up using some encryption key based on the user that that container user and that container user key only exists in that particular incarnation of the container image which means if you've got persistent storage and your config.xml is stored in that persistent storage it's going to have an encrypted string that a subsequent invocation of the container is not going to be able to decrypt at least that was my experience I'd relaunched a container it would just give me all these errors about I can't decrypt the password it wouldn't be able to log into the instance so um it's probably intuitive that using Mutual TLS which is kind of a better way to go it just has a few more moving parts so when I'm doing a quick and dirty experiment I generally don't go that far unless it's a mutual TLS related experiment um if using mtls that it's not an issue but if you're using just regular basic credentials which again I'm doing that I'm extracting those via Secrets manager and passing them as environment variables to The Container so they're not really exposed in the clear so from an exposure perspective that it was you know there wasn't a real compelling argument there at least not for my particular use case so so anyway that they provide you with this Force option that you can pass to the mid start command which is part of the entry point that gets launched by the container image off the shelf and so I just added that force and basically what that does is that just regenerates config.xml based on those environment variables every time it starts up which is like a negligible amount of kind of additional startup time uh so there was was really you know kind of an easy fix just leveraging the existing capability so then the other spot that needed tweaks was so for the mid container image there's this uh there's this asset directory and that includes supporting scripts it includes the script that the container build uses to download the payload if you don't have it embedded within your your build distro it will you can give it a URL and it'll just download that corresponding install file uh it's got a health check script the the kind of the key script that I had to touch was the a knit script which for Windows it's Powershell script uh here's where it kind of calls out the persistent directory where it can be like it where basically it will look for some persistent storage that excuse me that it can copy stuff to um in my experience it's it seems like it's kind of a feature a feature that's still being leveraged being kind of built on so like I said I saw it looked like it would copy the config file like on start and stop to and from that persistent storage but at least with windows that didn't that alone did not seem to work because of that kind of one time that ephemeral encryption key that would only last as long as the lifetime of that particular container so in order to leverage my persistent storage within fargate and the the fargate use case kind of the worst case scenario um and what I mean by that is if you're running in AWS they have containers as a service they call it ECS you can actually spin up your own nodes and if once you're spinning up your own nodes to host these containers you've got a lot more options with regard to making persistent storage available sort of so fargate if you can get to work with fargate you can get to work with basically anything else because anything else will give you a lot more control over things like your network traffic your ability to interactively go into the Container see what's going on and making storage available so what I did to make this work with fargate was I just went to the setup function which normally does things like that's what detects whether it needs to generate the config XML file based on um either environment variables that that also is what will pull things out of like if you've got things to find credentials defined in Docker Secrets or kubernetes secrets that that's the function within the init script that will generate the config XML based on the contents of those kind of those different options for storing and passing the sensitive and uh ephemeral bits of the configuration that you're passing into your your mid server so essentially what I did and again all those codes available in an example git repo so you can peruse it at your leisure so I'm not going to kind of dwell too much on the the minutia of what the code does but it basically just looks for my persistent directory and then it looks to see if the agent folder construct was already copied there and if it was then it'll just rebuild that Sim link it'll Create a Sim link remove the existing agent directory out of the image and then Sim link it over to the persistent agent hierarchy and again totally quick and dirty um in a real implementation you would kind of divide that up and um only put the stuff that you really want it to be you know read only just trying to increase the size so one of the one of the kind of benefits that I discovered is by doing it this way I can just Mount that same window share on a standard Windows box this is like my windows mid server my regular old windows mid server and I could see things like my containerized agent log by just navigating into that shared directory which is really great for debugging you know I don't know that there's a lot of compelling need to do that you know for full-on prod but if you've got lower environments that you do debugging in the fact that you can have it running in a container but also view what the container is doing without having to jump through hoops was kind of a plus for me so this is my this is my um FSX chair that is mounted here on Drive R and I can peruse my Powershell doesn't like dirt w i can peruse my logs I can you'll basically see everything that's running and all you know the literally everything that is present in the file system that my containerized mid is using so that was a big plus from a debugging perspective and yeah so um worthwhile exercise and and it got me more familiar with container recipes and like um any questions comments um want to recognize we're two minutes from the end of the kind of Standard time I don't know about you Mike today but I'm I mean I'm good to hang and talk as much as people have questions or whatever I do have a couple of other examples I can run through with CNO um you can dig into the the devops pipeline um Sabrina has her hand raised is that on purpose yes it is okay what's your question or comment well it's it's um I have a teammate who this would have been better for him unfortunately he had a prior engagement so I'm wondering is either is this recorded and we would be able to share it after or um maybe if he can reach out and ask his targeted questions can we have contact information yes to all uh it is it is recorded everybody who registered whether they attended or not we'll get a follow-up email with a link to the recording in YouTube um and um our contact info is in there as well so certainly anybody who has follow-up questions is welcome to reach out to me I'm not going to speak for Mike because he's my customer now so yeah that's up to that's up to you Mike but uh I'm I'm available for follow-up questions yeah as as needed that's great thank you so much this was a little above my um expertise as he would be the one doing it but it was also really helpful um what you've gone through yeah awesome thank you there's also Discord server so if we if you know we always usually send that out with the disc with the uh um invite as well those sorry the post meeting wrap up um so we people can jump on there and ask questions live of us and also the the other community members as well yeah good call out yeah the follow-up email will include kind of link to the it'll include the direct video link to this session as well as the YouTube playlist We maintain as well as uh the Discord server which we maintained so that people can kind of reach out out of band and since Mike's a customer now it's easier that's a kind of a an easy way for Mike and I to have a shared kind of a shared Landing Zone that we can both participate in yep great stuff other questions one thing to call out there is um well you had shown that health check script yep but that's what kubernetes uses to check the health of that container right to determine whether or not it's it's up and live and everything is working so um that is this is actually big and important right and understanding sort of like what's going on in here if things start crashing or it's not you know coming up and starting appropriately this is a good place to go and look yeah and you can actually see like when I look at my persistent share here it's got this DOT health check file which was updated uh well let's see if I look at it I think it's supposed to update every five minutes so yeah so it just got updated again at 4 59 so that means the health check script is running um and touching that file and then I think what happens is the health check script will look for that if it doesn't exist that's one condition that'll cause it to do an exit one and then it's just kind of a generic function within the container runtime that if it gets a health check return of non-zero it throws up an event saying like a failed health check and you can tell it like how many fails in a row have to occur before it will Mark the services or the the container is unhealthy right and um and that was yeah that was one thing with the fargate based experiment that I did was I noticed that the file share would go offline briefly sometimes and to be fair I didn't set up with any redundancy it was a lab I just kind of set it up real quick but when that happened the the current kind of experimental config I have was not happy it would just toss its cookies I mean the plus side is all I had to do is restart it and all the storage was still you know the persistent storage would allow it to pick up exactly where it left off including anything that it was trying to send to the ECC queue that wasn't able to before it died um and uh so in theory you could get that health check configured within fargate and then configured as a service within fargate which would automatically restart it when the health check failed there's probably other things I could have done to make the FSX uh Windows share more resilient um so it it seems imminently possible again this kind of all started because the customer is like hey can we run these in fargate because we'd really like to not have to manage our own container nodes we just want to spin up container images to do stuff and then make them go away yeah so it seems like a pretty good use case and just adding that ability to do persistent storage I'm sure Amazon will bring it along at some point but those timelines are indeterminate so in the meantime it gives people at least an option if that really is something that you know gets gives them value I mean to be fair spinning up a couple ec2s to be I mean you can you can actually create ECS clusters um with a default node site node count of zero so there's no workload you won't have nodes just sitting there doing nothing you'll your startup time when you do have a workload will potentially be a little longer because it'll have to spin up a node join it to the cluster and then it'll launch your workload but you know again it's like all of this stuff it's feeding all of what's important into like the the magic equation and coming up with what works best for you what's most important if the startup time isn't a big deal then you could you know not have to Monkey around with some of this stuff and just use native capabilities within the different uh the different providers so Varun had a question around certificate errors in a containerized midserver um Varun do you want to come off mute and talk a little bit about that that's something will and I might be familiar with yeah sure so uh what we are trying to do is we are trying to do a POC using a mid server and ACC in order to uh test out the servicenow discovery for the kubernetes resources like pods and state fruits sets and deployments yeah so um initially we did a customized image but uh there were multiple errors that we faced and then uh we use some other image provided uh like provided by uh from servicenow and after that uh I mean the mid server comes up but it throws a certificate related error so I mean we raised a ticket with uh support as well but nothing conclusive that uh that we can get so far so kind of stuck there and the pocs is also stuck is is are you using are you trying to use Mutual TLS for the authentication so it's a certain area for Mutual TLS or it's just talking when it tries to kind of talk to the to the instance yeah it is when it tries to talk to the instance that's when is it is your is your kubernetes environment behind a proxy oh so we we checked the connectivity from the kubernetes from the e-gas worker notes to the servicenow instance and the connectivities is there yeah but I've seen where there's like a proxy in the middle specifically I've seen API proxies kind of in the middle for outbound stuff and that will cause the the certificate that it's being returned back from the servicenow instance to appear not to match with the IP address that's doing the communications so you have this mismatch so um it's because it's literally a man in the middle type attack right so that might be something to look at okay sure um are you working with your account team on the servicenow side yeah yeah yeah I mean you can feel free to mention my name and say Hey you know they can reach out to me and I can uh look at it with them cert cert errors are um certificates are everywhere so it's kind of uh it's it's quite variable as far as what the um what the resolution is depending on where you know like Mike mentioned it could be if there's a proxy involved if the proxy is trying to do SSL decode then you'll get that then you could get that issue with um is it actually uh on like one of those unable to verify certificate type errors where it's yeah authenticated I pasted that error in the Q a yeah oh you did oh perfect and I do apologize I need to run guys um but as always it's really good um looking forward to catching up with you guys next month too thanks Mike take care everybody um let's see crypto error pure not authenticated okay yeah that's not a particular I've seen before but I'm certainly happy to kind of go back through all of my kind of container um container notes and see if I can find anything so yeah feel free to kind of mention my name to the um to your account team and they can kind of loop me in on the back end and happy to try and you know see if I can uh get you past this sure thank you any other questions I'm going to go ahead and turn the recording off because sometimes people are intimidated by being immortalized on YouTube