the recording we'll go ahead and get the process started thank you everyone for for joining today it's really lovely to see everyone um like will and I always say this is one of our favorite work days of the month um I'm gonna apologize in advance my voice sounds terrible um I think I have coid so I will probably um just kind of like help out with the intro here and then go on mute and Fade Into the background a little bit so um this is session 19 of beers with Engineers um and uh we always have to start off with a safe harbor statement right like a lot of what we're talking about is either net new features or um things that may potentially be coming out here so um take everything that you hear here with a grain of salt don't make any stock buying or overall purchase decisions based on any of this information um this is really mostly around us wanting to share and build a community with everybody um so this very you know informal agenda we always kind of run it like this um we we'll we'll keep it short sweeten to the point when it comes to the yaken and we'll get mostly into the tech stuff um this is a community thing so always you know um feel free to drop stuff in the chat or ask questions in Q&A or even you know jump in and ask questions this is as much for you guys as it is for us so let's talk about why we're here so this is the thing that will and I built um because we really felt like there's not enough Community conversation around the intersection between service now and Cloud native capabilities um and and in addition to that there's not really a great forum for folks to talk about um their experiences and share their experiences in that um in that vein so um we really built this for that and we love that so many people continue to show up and support us and uh um and even we had a great attendance at the live beers with engineers at knowledge this year so looking forward to doing that one again next year so who we are um this is normally the part where I introduce the beer that I'm going to drink I'm not going to drink beer today since I'm I'm Ill so I'm just going to have water um but I'm Mike Gallagher I am the manager of the Enterprise application platforms team at drw Holdings um I am a old school Tech head from way back and um have done just about everything in technology that you can possibly imagine um and most recently um I am actually helping run um a team that runs 14 different Enterprise platforms for a trading firm out of Chicago so I'm very much involved in um technology and how we use utilize it in order to drive business and and make it better big fan of kubernetes big fan of cloud native Technologies and I love me some automation so will hey everybody will ham uh I'm an advisory solution architect at service now I've been also doing technology for a lot of years currently focusing on it operations management and particular automating uh a lot of things in the cloud native Arena uh just really get a lot of satisfaction out of taking something that's cumbersome and repetitive and just taking it out of the hands of people and and just automating it my spare time I like to uh hang out with my family play pickup hockey and video games and today since it's October I will be drinking uh Samuel Adams October Fest cubot so uh Mike I think you said you're you're confirmed as well right you're going to be out Chicago so we will both be out in Chicago for cuon which is November 6 through the 9th and um there there isn't we didn't really get a lot of interest when we solicited for like a formal um sponsored event um so we'll just kind of be out there I'll probably be working at a service now Booth Mike will be just enjoying all of the uh Cloud native Innovations as a as a customer um but if anybody is going to be out there and wants to get together meet up just informally to chat about whatever um or let service now buy you a beer or two feel free to uh hit us up either via Discord via our email Alias um what whatever really quick I apologize chat is disabled um so if you have a question you come off mute or throw in the Q&A we'll jump into it with that I'm gonna go off video and go on mute and let will handle the rest of this I guess when I set this up as a webinar applied some template that disabled the the chat so yeah so as Mike said um I think everybody should be allowed to unmute um Yep looks like Mike's been doing a good job of uh catching people as they come in um so Leverage The the Q&A pan or feel free to just call come off mute and and speak up if you if you would like at any point okay so for this month I thought it would be cool to look at the uh new automation that we have around which basically integrates uh a capability that's been in kubernetes for a while which is the C manager um plug-in application module whatever you want to call it and the native service now certificate inventory and management function and um so I hadn't looked at it I I before um preparing for this session I saw some you know we got some kind of internal updates that this was coming to the platform and it it looked pretty cool um but I hadn't really been familiar with it until I thought it'd be a good topic so um just these slides mainly just tried to capture my experience with learning about it and setting it up and hopefully that helps uh other folks to um explore it in their own environment so a few kind of key points about it uh as I mentioned it leverages the C manager Kate's application which is I forget where it's ranked but it's pretty high up there in the cloud native Computing Foundation um project activity rating uh you know so it's it's certainly nothing new Under the Sun at this point it's pretty robust in fact it's a dependency for one of our other products the service graph connector for open Telemetry that has a dependency on CER manager to manage the certificates that are required for the open Telemetry collector to receive Telemetry within your cluster so just an example of a one of many tools which have a dependency on C manager so it's very likely that's a facility that's running inside your own kubernetes clusters today and so what we kind of use to connect C manager with service now certificate inventory and management is we created a recipe for creating what's called an external issuer and it's basically just that's one of the hooks that se manager provides you can Define external issuers in your cluster which are basically end points that c manager can use to request certificates because that's its function is it requests renews revokes certificates within the kubernetes cluster so the first thing we did we created a recipe for uh which can go into a manifest to spin up a pod which runs this executable which on the one hand ingests these certificate requests via C manager and then communicates with your service now instance to plug in uh Certificate request tasks onto the certificate inventory and management functionality and then we also produced uh a Helm chart as well as a raw yaml manifest which can be used once you've got that external issue or image to deploy that pod into your kubernetes clusters and integrate it with the the CT manager that's running there and once you've got those pieces in place you just submit you know a Certificate request into your cluster via whatever standard methods are used within your organization and that can be just a manifest which specifies a certificate or cicd pipeline really you know anything that kind of gets that payload into your kubernetes cluster will then kind of trigger the automation which then submits a request for a certificate on the service now side which can then be automatically fulfilled and it monitors those tasks on a configurable interval when it sees the task is completed then it pulls the certificate down down and installs it into your cluster in the form of a secret and this works with from what I could tell so certificate Inventory management has two kind of paths it's got um it's got a manual certificate fulfillment path which can be used if you are either you know not ready to automate or you're just using a CA or or some Casa which for which we don't have out of the box automation uh from what I could tell this only goes it feeds into the automated workflow so um it does require that the ca from which it's requesting a new certificate is one of the ones that we support for automation um that list seems to grow every time we come out with a new store release the Microsoft CA was not there initially and now that is in place that's what I used for my little test setup since as best I can tell none of the other CA we support have like a free or a community option and I feel like uh trying to expense a bunch of um you know GoDaddy CTS or or whatever uh lessons learn this is where I usually capture gotas that I ran into when I deployed my little test setup um pitfalls I ran into and really there there weren't a whole lot um the bulk of the issues I ran into were just due to the fact that I'm not a Windows admin by trade had not set up a Windows CA before and so I was setting one up from scratch and um the so the really the only thing that was a bit of a stumbling block was um for the windows CA specifically the subflow which performs the certificate fulfillment is the the actions that actually do the the heavy lifting our Powershell actions and they require uh what's called cred SSP which is specific kind of a credential authenication um format protocol within windows and that seemed to be a little bit fussier about the trust between my mid server and my CA um so I did have to kind of do a little Windows searching to make sure that all my ducks were in a row for cred SSP support to work uh there are some pretty good um diagnostic commands that you can use to kind of validate that that connectivity works so between that and just like running the flow a bunch of times and observing the the Powershell output I was able to work through that pretty pretty quickly and I would think anybody who's uh a Windows kind of full-time Windows engineer would probably have a much easier time than than I did uh the other thing that was just kind of weird was for whatever reason um I it was difficult for me to find the download page that contains the the collateral that I that I talked about and that I'll be showing you in a couple minutes which contain the image recipe and the helm chart um and I'll kind of illustrate what I mean when I uh when I do the demo um just going to answer this question of venafi integration I'll make a note of that and I'll see I'll check with the product team and then I'll post the answer when I send the follow-up emails um with which include the slide deck I don't know what's on the road map um for New U Casa off off the top of my head but I will take a note and follow up on that another point to add there is if you'd like to it's worthwhile putting an idea in the idea portal on the community um because it could potentially get upvoted enough to become an enhancement that will make its way into the product long term yeah that's a great Point um the idea portal oh darn it I'm blanking on whether it's a sub subsite off of the community un or off of support at this point because it's kind of moved around um but I'm going into the demo anyway so I can take a quick look I I think I think I have the link here yeah I feel like it's a top level I thought it was a top level yeah so if you go to service now Community under resources idea portal um this is kind of our official channel for voice of the customer where if uh you know there's a piece of functionality that's not there and would be valuable a customer can either kind of look in here and see so I don't see any requests um live on the portal right now for the venafi uh CA so that would be one Avenue I I generally recommend folks check the idea portal and also just make the um you know make make the fact that you're looking for that known to your account team because they can generally funnel it back as well to the product team um I think critical mass for ideas is 10 if it gets 10 up votes then it kind of goes to the next stage of kind of further consideration within the product team okay so in terms of uh demonstration I just figured I would show how I put all these pieces together and then just give you a quick demonstration of the functionality in action so what I meant in my uh my lessons learn slide when I talked about getting to the download page I don't know whether I just kept missing it in the docs or or what it was but uh it just was not clear just starting from scratch that the way you get to the download page which contains the docker recipe um it also contains a static binary that you can just copy into your uh like if you don't want to have the the docker image build download the binary on the Fly you can obtain the binary for the external issuer from this link and then just incorporate it into the directly into the assets directory for your your image build and then it's got the helm chart as well as a raw ketti uh am but getting to this this is actually available via a workspace so if you look at workspaces certificate management and then this is Tokyo and so on Tokyo the way the download appears is over here and then just for like by way of comparison this is how it looks when you go to that same workpace on Vancouver so they kind of shifted it around a little bit and so now there's a downloads like a downloads tab I guess I would call it so just something to be aware of it took me you know half an hour or whatever of kind of poking around to find this so if I can save somebody else that that time then it is goodness so to put my little setup together I started by downloading the docker recipe and then because I hate doing manual Docker image builds even one time um what I did was I just created a very simple just pulled it down to my Linux machine here created a very simple build spec so I could build it using AWS code build um this is you know this it's super straightforward and if you just do it manually it's not the end of the world either it's basically just doing a Docker build and um then tagging it and uploading it to my private repo so that all went pretty smoothly no major issues um you know it is kind of another build it-yourself image we don't have a publicly serviced image for this at the moment again it's kind of facing those same challenges where it's not um it's not a minimized image it's it is based on a Linux drro and because Linux distros are constantly getting flagged with um you know vulnerabilities and and container images especially get scrutinized for that the the decision was made to kind of bring this to Market quickly by just putting that in the hands of the customer so that if there's a standard Linux drro that you already have that's minimized and hardened and whatnot you can just customize the image to be based on that and bypass a lot of um you know potential security vulnerability response type um type of rigor around generating this image so once I had my image uploaded to my private repo then I moved to over to this uh this right side of the download page pulled down the helm chart hold it on to my Linux system and then what you do is you just customize this values yaml file to match up with your environment so the key points there um if you do have like namespace standards or or what have you where you don't want it to go into a default namespace you can specify namespace here uh it's definitely obligatory that you put in the appropriate service now instance that you're going to Target and and I mean actually you don't have to you can set specify these that's the nice thing about um about Helm is you can also override these on the command line if you prefer so you don't have to have a static values file if you want to work this into a uh you know your cluster build um cicd pipeline for example you can you know plug it in with environment variables or you know override it on the command line um and so we basically set a few basic parameters like the environment the certificate purpose uh how long you want the certificate to be valid for uh you can insert certificate owner group and certificate owner if you want and then you'd give it a time interval on on which it um will monitor the new certificate task table I think by default it's set to I think I think by default it wants to do 15 minutes I just turned it up to one minute because I was specifically putting this together for a a kind of a dynamic demo I didn't want to be sitting around for 15 minutes waiting for the next step to kick in um and then down at the bottom here you plug in your private repository and then you can specify you know if you want to um apply overrides to how often the image is pulled uh if there's specific tags to use there's a place where you can put in um image pull Secrets over ride the uh name values resource the rest of it's fairly standard the main you know the the main key things you definitely have to put in there are your instance name and your your repo and then I deployed that Helm chart to my cluster and what I end up with is is this pod running in the SNT management namespace and it's named SN issuer controller manager and then so there's a pre there is a preck and we do those instructions are included here um if you don't already have C manager running in your cluster to support you know because like I said it's kind of the standard when it comes to automated CT management within kubernetes so um chances are you may already have this running in your environment but if you don't you do need there's uh you know Helm charts and other and pretty much um you name it there's probably uh a payload to fit that kind of kubernetes deployment facility to deploy C manager and so I've got my C manager running and and then I've got my external issuer running as well so one of the things the helm chart does is it populates the hooks so that c manager knows that when it gets a Certificate request that calls back to this particular external issuer that it you know it knows okay that's where I need to send this particular certificate Certificate request uh just looking at the Q&A for a second um okay Johan says hi to Kim I guess that would have been I'm gonna have to check my settings and uh I didn't purposely disable chat and it it would have been useful to have chat enabled for that kind of friendly inter action so I'm going to have to check that for the following sessions and um let's see ja car is asking is there documentation stating the workflow flows within C SE now forer manager what is exactly is happening in the background um I don't know I I let me I'll make a note to see exactly how explicitly it's documented um it is all executed via flows so if you go to uh if you go to flow [Music] designer yeah it's a good thing that this is all implemented with flows it would have been a real pain to try and uh work through my my my setup um my setup speed bumps on this window Windows CA if I had to debug a bunch of JavaScript code directly so if I filter out just show the C Inventory management subflows so they're all it's all right here so for example for the Microsoft CT fulfillment there's a subflow that kind of handles all of the operations and then there are custom actions that are linked off of these subflows that you can kind of follow along and open the corresponding Uh custom actions within the flow but then if I look at my executions I can see you know I can see an execution for each of the requests that I ran through my test and there um especially since this it's nice because with the Microsoft One it uses Powershell so you can turn on Powershell debugging at the platform level and then uh can produce some very um very verb Bose output from the Powershell command uh there's actually a uh there is actually a knowledge KB that let me just make a note I'm going to include that in the links when I send out these slides because there is a little bit of a quirk when it comes to enabling Powershell debugging for flows where um there is kind of a special process for enabling that and and it's only something you're going to want to enable when you're debugging a particular issue because it actually what it does is it inserts the verbose error output with the rest of the output of the Powershell command which will invariably break any Powershell um any parsing that you have in your in your Powershell action where it's parsing the output I discovered that uh through this through this exercise uh poers shell debug maybe um but I will check and see if there's any formal documentation off the top of my head I think I had to kind of dig into these flows on my own to kind of see what was going on but they're not really obscured they're all very kind of um you know intuitively named and and you can just look at exactly what they're doing either after the fact or ahead of time if you want to kind of review it up front um in theory you could do a test run although some of these some of the inputs that it's expecting um might not um might not be easy to kind of put together as a test payload but other than that it's uh perfectly it's a perfectly viable approach um so the rest of jaar's question external issuer is sending tasks to instance on what basis is all calculation being done um it sends it it will base it on the spec that goes into the um the goes into your Certificate request that goes into the cluster so what I mean by that if I go over here here's just a simple example Certificate request for kubernetes so the the you know the scenario would be if you're running a pod that's provisioning uh you know or M or orchestrating a web service and you want that web service to use TLS this lets you automate the process of provisioning a certificate for that web service um so what kind of determines that it gets sent to the service now external issuer is this issuer ref section here and what that does is when the helm chart Provisions the external issuer it applies a name and a group to it and so it's simp SAR to a it's similar to like a selection criteria for running a pod this has to match up and based on the name and group matching C manager will submit that request with the rest of the parameters the subject and in this case the DNS names the subject uh it sticks the common name in there uh the locality information the course responding email address duration in the renewal window and then the you know as we get to the kind of U functional demonstration you'll see where it'll basically create a task within this new certificate task table and then periodically monitor you know basically makes an API call to submit the task and then it goes into a loop where it will monitor that task for status and what I observed in my multiple failures that I incurred upon kind of learning how to do this is eventually it does detect a failure and eventually the external issuer will resubmit the Certificate request I didn't didn't see I'm not sure what the interval is on how often it wasn't often enough for me to uh do meaningful testing so I generally was just deleting the test CT request and and resubmitting it to get it to to redo but there does appear to be some retry logic built in there does detect that the uh if a task fails to uh complete okay so um it was a good segue because the next step once you've got all these pieces in place is to attempt to request the certificate so the way I'll do that is I'll just apply this simple manifest so now if I I look um I look at my cluster so I'll see that there's a certificate record object named with the name matching what's in my uh what's in my manifest it'll show a status of false because it hasn't been provisioned yet and it'll show the corresponding secret which once the certificate does get created that's where the certificate will end up residing so now if I go to my service now instance and I look at my list of new certificate task I can see there's a new task and the subject common name matches what I had in my example manifest so um in order for in order to do something with one of these tasks you do need to create what's called a routing policy within certificate inventory and management um these can be either fully automated or include uh you know whatever kind of approval chain you want so I set mine up to have an approval chain just so I can kind of validate everything and not spam my uh my instance which a bunch of you know failing flows if I did have some kind of an issue so um since I was using us a Microsoft CA it uses Powershell so this is a Powershell credential that I created which can talk to um or it's a yeah it's a Windows it's a Windows credential um so that my mid server can communicate with my CA and make the required Powershell commandlet calls and then I defined this um here's my CA record that I created just kind of a stub just so that it recognizes the name whoops never Ed the never use the service now back button when you've got multiple tabs I said you have to provide an approval group an assign or an assignment group for the uh regard even if you're doing full uh automation it does need to have an assignment group for the routing policy here's where I identify the windows certificate template I just put together a a quick test template for these requests feeding into the windows CA here's where you specify the host IP for the windows CA subject common name um um what else is pertinent I did you know I've got multiple kind of different environment mid server set up so I um did specifically point it at my appropriate mid server that's part of the active directory domain that the ca belongs in you do have to assign a task approval group this is PR this is just standard C inventory and management this is there's nothing unique here by virtue of the fact that these requests are coming in through C manager from a kubernetes cluster so now that I've got that routing policy all set up um again this is kind of I went kind of semi-automated with this so it does require approval but then once it gets approval then the rest of it automated but that's completely user configurable and so these approvals have a special button called choose routing policy and approve so because my routing policy um you can you can align a routing policy so that the system just picks a routing policy based on the um the certificate the values in the Certificate request so I I just didn't do that I wanted to kind of keep it keep all the steps visible but so this is not a necessary if full automation desired I just left it as a step along the way to allow me to kind of Walk The Narrative through because honestly if I didn't it would be fulfilled before I had time to even describe what was going on so now I'll apply the routing policy and so now that task is approved D and if the if the gods of live demo are with me today which they are I now have a completed task with the uh certificate attached then if I go back to my cluster uh in practice it does seem to take a minute or two for so basically what happens is the task is completed the collateral the the certificate itself and I think this is this would probably be the ca chain C are attached and then we have to wait for the external issuer to kind of loop back through check the status of the task and feed it back to CT manager um so generally that's going to be you know a couple couple of minutes uh in the meanwhile we can look at the execution let's see that would be this one so we can look at the execution of the flow and see all the steps that were performed fairly straightforward logic it checks to see what type of request it is and then routes the logic accordingly so here's the custom step to request renew Microsoft C you can see the individual Powershell steps executing it you know it enumerates the output from each of them so this was really handy when I was working through some kind of cred SSP um access and authentication issues and that kind of stuff um but the fact that it's implemented via subflows lends itself to you know debugging I mean even if you know there's a a required requirement for customization particular to your environment it's all kind of very uh transparent and straightforward so we're checking that we received the certificate with no error then we're looking up the C management task updating the C management task and then we're running this action to populate the certificate and attachments there some error handling down here but we had no errors so that was the end of the subflow let's take another look at the cluster and so now we see that that certificate is listed as ready and so if I go and look at this secret now because that's the kind of the vehicle by which kubernetes services are able to retrieve and use that certificate and so here's my base 64 encoded certificate here's the the key and so it's basically you know standard format consumable inestable by any number of services running within my kubernetes [Music] um kubernetes cluster uh just going to the Q&A for a second um that ja car again that flows for the Microsoft CA right not exactly for Kate CER manager do we have to make our own wasn't able to find well so the C manager piece that is you don't have to that's not something you have to create that a a standard kubernetes um and actually I'll make a note because I'll include the um actually that link is probably here so if I go to the build instructions I think that talks about how to deploy CT [Music] manager no I guess it doesn't just just listed as the pr inst yeah so search manager by default when you install it right there's capabilities inside of it for it to go and request new certificates and by deploying this Helm chart for the service now piece um it becomes a new issuer type that gets registered within the cluster and then when you request a certificate with that issuer Ty certain manager knows to go and look for the service now issuer um in order to issue that certificate yeah so as far as the CT manager is concerned you that's not something that you have to create um if it's not already in your environment you would just hit C manager doio and pull in whichever the you know Helm is is pretty popular so there's a Helm chart for deploying C manager so it's as simple as just Helm install CT manager and there's no additional coding or configuration required for that piece and then you know like Mike said the the helm chart that deploys the external issuer for service now again just kind of sets all that up so all you have to do to make sure that a Certificate request gets routed the service now is when you put together a spec for a certificate within the kubernetes cluster is that you craft this issuer ref with these values that's that that's that's the extent of the I would I would hesitate to even call it customization it's really just uh an aspect of the Certificate request that just needs to be specified in that way in order to cause the CT manager to then pass the request to our external issuer which then sends it into your service now instance and and actually will if I can steal the screen here for a second yeah I have a diagram of the flow of how it works on the back end now go right ahead um see here oops so effectively what this does right it creates the new Certificate request and then based on on that it determines whether or not it needs approval um and if it is it goes over and creates the automated task um using these tables and then there's a business rule here it's called out um and a script include that's called out that say that looks at it and says okay does this match an existing routing policy and if it does then it triggers the routing policy um and and and then gets the chain back and attaches it to the task and then and then pulls it back in from the CA and and drops it into the uh the CI table um if it can't figure it out that's what will was showing it generates that approval record which says hey not only do you need to approve this but you also need to figure out what the routing policy is um and then if you approve it and apply the routing policy then it follows that same piece otherwise if not it just goes to the end and says no I'm done those things so um this is how all of the automated approvals excuse me all of the automated certificate management workflow works on the back end out of the box it just so happens that this first step is being triggered by the kubernetes certificate issuer that's been deployed in the cluster and then the last piece is it picking it up from the service now instance off of that um task and putting it back into the kubernetes cluster as the resource that you requested it sorry I'll stop sharing oh yeah looks like uh Johannes um put his idea in there for the vfi um it's in the uh Q&A so if everybody would go and upvote that to help out that'd be great you will um yeah so and then just to uh just to reiterate because it is kind of the important where that linkage takes place um this section here is what kind of routes that Certificate request from your kubernetes cluster into your service now instance and then I didn't explicitly call it out it is it is in the instructions for setting this up that knowledge KB uh this knowledge KB here um you basically create a kubernetes secret which um is used by the where you plug it so you create a you create a user that has these um these two standard CT management excuse me CT management roles on your instance and then you create a kubernetes secret that corresponds to that which is used by the helm chart to authenticate and make the API calls and the instructions for that are in this KB you kind of see down here it specifically gives you the the instructions and then there's an example Certificate request this is probably what I use to generate mine and again that's kind of the key thing here this is what sends it over to us as opposed to the standard kind of CT management um that CT manager does out of the box any other questions okay all right I'm say goodbye to uh all our viewers out in YouTube land stop the recording and so now we're at the part of the program where we just open it up for any kind of discussion if anybody has any topics that they want to bring up um for this month the floor is open if anyone has any questions topics comments Etc going once few questions so far today during the session appreciate that yeah absolutely that's really what I think that's what kind of makes these webinars is the fact that people aren't shy about asking questions and um and providing feedback hopefully you know we [Music] can get some uh legs behind the idea of uh venafi integration yeah jerich car you you got a question feel free to come off mut yeah here hi Mike hi Will so it was an interesting session on the kubernetes and s Management stuff I'd like to share my understanding of this whole process of s Management sure so it it all initiates from the discovery part you get your Discovery stuff done under the SE management certificates are discovered uh by certificates I mean kubernetes Sears so once you have them on your unique search table or insert of service now or then uh you uh you start start this uh s manager uh you you start using the SE manager which is for automating the approval which is renewing revoking or uh doing stuff around operational stuff around certificates right so I have a lot of questions related to the U interal processes of of this eques SE manager in particular so um we have the certificates so you said that flows are for uh Microsoft CA not for the KU resert manager which is quite understandable I guess so my question is uh how can we exactly validate first of all this uh kubernetes so after deploying external issuer and the SE manager uh we we tried using the the demo file which will for obvious reasons it it won't work right so even if we so suppose we have the EML file with with all the attributes and uh we do the cube CTL apply f with with that file so what does exactly kick in the back end that external issu it and create to create a record in in service now on what basis is it going to get rejected or approved or failed oh like what is happening what is exactly happening in the back end so is it is it supposed to be defined inside of the flows or is well what you should see is when you if you submit uh so if you submit a yaml file similar to you know similar to this example or the example that I used y uh what you should see on your cluster um is first you should see you should see a certificate show up in a not ready state that would show you that the cluster has ingested your request for a certificate um then what you should see the kind of the next step would be you should see a task show up on your service now instance uh under the new certificate task table which corresponds to that common name that you put in your yaml file if you don't then you've got your then you do have the ability to kind of potentially look at what's going on with within um I guess I would start with the external issuer so you could do and you may need um you know you may need to engage your your kubernetes admin team to do this piece you can look on your cluster or they can look on on the cluster and pull the logs so this is just example of logs that are coming in from our external issuer and this would show you not sharing your screen if you're trying to show stuff that's weird am I sharing screen one I was sharing screen one you would see so basically just to reiterate I ran a logs command against our external issuer pod and so this is just dumping out all of the logs um when it stops there we go you can kind of see examples where it's saying okay I recognized somebody submitted you know CT manager has passed along aert request to me and so you would look you know presumably if you're not seeing a task show up on your service now instance there would be an error indicating that there was some kind of a issue contacting the service nown platform form from the external issuer uh if you don't see any indication here that you know that it's generating tasks or that it even received a request for a certificate then you could have your kubernetes team go and look at the CT uh the CT manager pods themselves uh and do the same kind of the same kind of thing pull the log from the C manager pods um oh sorry copy paste error so you know presumably if there's an issue submitting a Certificate request to the external issuer this podlog from C manager would give you some kind of a you know some kind of an indication but again that would be um you know if if you're It's Not Unusual for service now platform admins platform owners to not necessarily have full access to the kubernetes Clusters that they're discovering that they're supporting so you may have to engage you know whatever team manages your kubernetes clusters to get to this level of debugging but really from a kind of a flow perspective when it comes to provisioning or renewing certificates within a kubernetes cluster it doesn't become visible on the service now platform until the external issuer submits that standard new certificate task hopefully that helped clear it up at least somewhat uh the other thing I would say is it's not required that you discover certificates before you try and use the provisioning capability um it doesn't it doesn't hurt but they're not kind of uh there there isn't uh there isn't a hard and fast require what'll happen is if the CT already exists and it just requests it by virtue of going through this process it'll get populated in the cmdb but you can certainly um you know if aert already exists then what I've seen because I've kind of repeated multiple CT requests without um changing the you changing the C name what I found was if it's already in the cdb it'll just send it right back immediately it won't send it for new um you know it won't need to submit it to the ca at all if it's already in the cmdb so it's certainly useful to have discovered uh as many of the C CTS that exist in your environment as as you're able to so is it uh is it applicable to say that this is uh still not uh fully automated stuff and and uh like like we still manually executing that command to to generate the certificate task and doing some stuff around it the level well what is fully automated is once once the certificate is requested within the cluster then it can be fully automated kicking off that you know submitting that yaml file the way I did um the only reason that was manual is for demonstration purposes if there's for example if there's a cicd pipeline that is deploying this application to kubernetes part of that cicd pipeline could be to submit that request submit include the certificate as part of that yaml manifest that's getting or Helm chart that's getting deployed to kubernetes in which case once that certificate object appears that's kind of that's the trigger so it's perfectly conceivable within a customer environment for the whole thing to be fully automated it's not um but as far as what service now provides service now provides everything after this initial Certificate request is added to the cluster so that piece could be automated but that's there's no standard way way to do that every customer is going to have a different slightly different at least method for submitting certaint requests into their environment whether it's part of their cicd pipeline potentially it could be part of a separate you know and there could be another customized service now workflow that's doing it all just depends on what your model is for deploying applications into your kubernetes environments I threw in the chat uh like a quick guide on CT manager um and it gives a good pretty good overview of kind of how it works it's also important to note that CT manager will issue requests for expiring Sears too so if you're using certificates that are signed within your cluster and they are coming up on their expiration date C manager will actually also do a Certificate request resource to renew that certificate as well and that whole process is is fully automated um and configurable and um and then once it gets into the service now platform you know what what we'll show today was kind of the more manual flow requiring approval um but you can certainly build out a routing policy within service now that um does not require any approval or any human intervention whatsoever um that's certainly possible um that just wasn't part of the demonstration today yeah that's a good call out Mike that's one of the benefits of cert manager is once you've got aert in your system within your kubernetes cluster it keeps track of them and when they're about to expire it'll automatically reprovision and and renew them yeah so it it did give a quite Insight quite a lot of insight into into the process I'm still having a lot of questions so do we have capability or resources to take it up or yeah I would say start with your um start with your service now accounting um there are there's going to be some links in the when I send out the slides there's some links um there are a couple KBS about the external issuer um but I would say um on top of that leverage your service now account team they can get kind of backend resources within service now and put you in touch with them to kind of look into your specific environment give you recommendations and um kind of you know at least give you some some guidance on where to you know on on where to go from here um and then yeah th those are kind of the uh well and then the other thing is a lot of customers are leveraging you know Partners so certainly um you know if if you happen to be using a partner for any part of your kind of managing your your instances your platform they're definitely a resource to tap into as well but yeah I mean that generally the go-to is um you know talk to your account team I basically to put this together I kind of reached back to the product team to get some collateral and some um some of the initial kind of transfer of information content that was put together so I would say if you you know talk to your talk to your service now accounting they should have access to at least that same kind of stuff that they can um provide to you sorry I had to step away for a second did we did we I was just uh I was just saying definitely leverage oh leverage that service now accounting to when you need more kind of either internal resources or just more kind of detailed information about really any of the platform capabilities yep did we dive into um the question in the Q&A right now oh no thank you for uh building service Maps is Duro wants to know if we covered building service maps from ISO using CNO in any of our previous webinars um we may have touched on it in one of the because we've done a couple C sessions um but I don't know do you do you guys have a lot of IO within your clusters Durga uh yes Mike we are planning to have the sto Service smes uh controller um in eks so we just want to understand we we we took your help I really appreciate your help in you know operationalizing of CN for visibility purpose so like the feature um and now we the next logical strategic point is to build the service Maps using sto service map so that's the reason the background of that question was know was there any webinar or any kind of a session we did in the previous Bas with Engineers yeah I don't think I I we probably touched on it but I don't think we went super deep yeah let I just made a note I'm gonna follow up on that because I haven't seen it explicitly spelled out road map wise but I'm not sure what the longevity is going to be for stto just because it didn't seem to really get a lot of adoption within the customer base and due to the um the challenges with maintaining um container images right now the Strategic direction for CNO it's funny you should kind of bring this up because I just um we've got a new uh lightweight publicly available CNO container image coming out with the November store release that um is kind of going the other other direction where it's going very lightweight and coupled with the fact that we released the open Telemetry um service graph connector which does a lot to generate service Maps based on Services talking to each other within the cluster um there's you know I I don't want to I don't want to uh read too much into it but it does seem like it's at least worth asking the question whether what the future is for the stto piece of the functionality because it definitely hasn't come up lately internally so I don't know in the CNO product uh um information page they specifically called out that um they can we in through CNO we can by importing um service mesh data yes build the service Maps want to understand that anywhere realized or what is the is there any set instructions or anywhere guide or somewhere so that we just want to do a POC and see what kind of data it can reveal so we can take it operationalization I I did it um when it first went GA and um I'll I'll kind of go through my notes to get the specifics but what my just off the top of my head what I recall was um you put you put ISO on the cluster you tie it into Prometheus and then you include when you discover your cluster you include the Prometheus URL in the pattern um in the in the pattern parameters and then when you do that it pulls in the network flows that ISO is recording correct and um one of the things that I ran into was that IO was not properly reporting into to Prometheus so I was running into issues getting that flow data in where I needed it to be able to build those service Maps so that's um that was a configuration issue on my end right um but something to be aware of is that because it it actually pulls that flow data from Prometheus um you you have to have that Prometheus report sto reporting into Prometheus appropriately in order for that to work right yeah I had the same issue with my because I started out I had kind of like an all-purpose Prometheus instance within my kubernetes cluster yep and that wasn't seeing anything and so just to kind of make it work I think I ended up using ISO CTL to install the iso specific Prometheus Plugin or or yep deployment and and it worked in that very kind of specific targeted um deployment I'm sure there's a way to make ISO talk to a more kind of cluster wide Prometheus um facility but I was just kind of doing some quick and dirty experimentation so I didn't kind of Follow That to its logical conclusion once I got it once I got it working in some fashion but um yeah I think I've got some notes on it or you know I'll kind of go back and and and see what um what I can retrieve from um when I was when I was looking at that it wasn't that that was the only that's the only kind of Hiccup that I can recall other than that I was mainly um I was leveraging the stto CTL utility to kind of configure all the iso stuff within within my one cluster so I didn't have to you know dig in and and individually manipulate different uh different components too much but um that's that's that's what I remember I'll follow up and include uh what my findings when we send out our our wrapup emails thank you will appreciate sounds good good stuff okay Q&A is empty does anybody have anything else okay all right well thank you as always everybody for the the engagement and the the Lively uh the Lively conversation Mike thanks for braving not feeling well worries sorry looking forward to next week when I feel better yes all right so our next session's uh mid november-ish the 16th we're gonna be at cubon like about a week before that so if we don't see any of you at cucon in Chicago uh hopefully we'll we'll see you next month uh later in the month at the uh the next webinar thanks everybody thanks Mike take care guys feel better Mike