right so as usual we are recording this session um everybody welcome to the beers with cloud engineers session four um today is june 23rd and uh we are gonna be talking through a whole bunch of stuff that is forward-looking statements that is kind of you know deep dive into the product and technology so don't go out and make any stock purchasing decisions or anything like that based on any conversations that we have today this is just purely informational session only and again super short agenda we try not to be too formal here this is really more about the technology and and bringing you guys up to speed so i will highlight kind of why we're here who we are and then we'll dive into the the cloud api updates which is the primary conversation for today but we also have a little bit of a conversation we want to add in around some of the change capabilities within uh event driven discovery that we're looking at as well all right so why are we here the goal of this is always to make sure we're bringing together you guys to talk about where you're at in your cloud native journey and how you're utilizing servicenow to manage that cloud native technology portfolio and kind of where you're going to next the big portion of that is us talking about hey here's the new technologies and the new capabilities that we're releasing uh but a lot of it also is just conversations around you know what you guys are doing and what you're having success with so who we are if you don't know yet if this is your first time joining us my name is mike gallagher i'm an advisory solution architect uh here at servicenow i've been in it for far longer than most people would assume but uh 25 plus years i've done everything under the sun but really now my primary focus is around cloud native technologies and um you know how it all plays with servicenow i think a lot about kubernetes a lot about containerization and where we go i as i tell everybody i have a propeller head and i love to use it to solve business problems will thanks mike welcome everyone my name is will hallam i'm an advisory solution architect for itom at servicenow i've been here just over a year before that i spent about six years working at a servicenow customer where i found servicenow was a great platform to help me automate a lot of cloud management and on-prem infrastructure management activities and it's been exciting to kind of show other people ways to do that using our platform and my spare time i love to hang out with my family play a little pickup hockey all right so let's get into the the meat of why we're here today right the whole goal today is uh as of 1.0.90 of the discovery and service mapping patterns we've released the ability now to essentially do automated discovery of kubernetes clusters against gcp and aws and in july which we'll kind of dig into a little bit more azure will be an option as well and the whole goal behind this and we'll we'll dig more into it in a minute here is now you won't have to manually create schedules serverless discovery patterns and credentials for every single kubernetes cluster right it'll be hey you can figure out the appropriate things for the cloud provider itself and then everything else kind of takes care of that beyond that so um that being said i'll go ahead and let will kind of dive in from here awesome thank you mike so as mike alluded to we've got now got support for automatic plus um kubernetes clustered discovery schedule creation driven out of the various cloud providers kubernetes as a service so for those of you who haven't had the enjoyment of discovering kubernetes clusters up until now it um involves a series of steps for each cluster traditionally and and so for one or two clusters that that wasn't really a a big barrier to entry but the feedback that we got and and that we encountered ourselves if we tried to scale that up in our in our lab environments was it can get a little tedious to have to go through those steps over and over again and so uh based on that feedback our pattern development team has put together enhancements to the existing discovery and discovery patterns store application so that was made available in may and as mike alluded to there's an update coming in july i believe it's late july around the third week of july which uh kind of rounds out the the power trio of cloud providers so today we're able to as just as part of native uh google cloud and aws cloud discovery detect when there are kubernetes as a service clusters and so what that means is if you're in aws or refers to eks and if you're in gcp refers to gks if there are kubernetes clustered provisioned in a cloud account which is being discovered uh some basic table entries are created in the kubernetes cluster cmdb table and then after your cloud discovery completes or after that pattern that pattern completes within your cloud discovery a postscript runs which then goes through those clusters that exist in that kubernetes cluster cmdb table and generate a discovery schedule um for each one of them and the need for credentials is also um deprecated for that particular flow because there are cloud api calls which can dynamically provide temporary credentials that discovery can then use to discover the cluster and so what's happened is there's code behind the scenes that is now pulling out temporary credentials using the appropriate cloud api call and feeding those into the standard kubernetes discovery pattern which then runs against those clusters um so in this uh in the deck you'll see at the bottom of that last slide there's a uh and we'll send these slides out after the after the webinar there is a knowledge base article on support.servicenow.com which should be visible to all customers at this point and that's a kind of a critical piece that currently is like basically errata from the documentation where it fills in a few blanks they're currently there in the stock documentation that talks about setting this up and that knowledge based article is pretty pivotal because it also includes an update set that has to be applied to any instances that are not running tokyo or newer starting with tokyo the requisite global objects will be included but since this was rolled out with support for rome and up if you're running rome or san diego then there's an update set that needs to be applied in order for this discovery mechanism to function correctly all right next slide so the next two slides are really just for reference um we've captured the different pieces that either were added or modified in order to support this and we just felt it was kind of important to do that because this is kind of a technical you know this is a technical forum and we just wanted to kind of provide transparency and also um you know if you're trying to set this up and you're running into some kind of a hiccup or an issue these next few slides show you kind of all the pieces that are involved all the moving parts and i kind of had to discover this myself as i was kind of setting it up for the first time and working through a few you know a few hiccups and so this is uh this is a really useful list to have because it kind of follows the whole process and shows you every piece of content in your instance which is doing a piece of this functionality yeah the key point there the mid server script includes that's where it actually goes and runs the credentials gathering process for each of the cloud providers when it does the kubernetes discovery so you'll notice that it has requirements in there for the aws cli to be installed on your mid server as well as the azure cli or the um gcloud cli if depending upon which cloud provider that you're on so that's a key point to be aware of and understand that those groups are actually getting pushed down into the mid server and they're executing locally on the mid server so that they have the necessary permissions okay let me clear out my little circle there and um you know what i got so caught up and excited about automatic um kubernetes discovery i i forgot to open my frosty beverage for the day which is uh uh daisy cutter pale ale from the half acre brewery in uh in chicago so i'm gonna just kind of open that up here um also i was remiss in popping up our uh our introductory poll just to kind of get a get a sense of the room it's got a couple basic questions just trying to gauge where everybody's at in terms of their journey to cloud native specifically related to kubernetes um it's just uh that's i mean selfishly that's kind of one of the reasons we started this series was just to get more in touch with servicenow customers who are you know somewhere in the journey to cloud native and find out what they're doing what matters to them what's difficult what's easy what lessons have they learned so that we can try and you know use that feedback loop and feed it all back into the platform and and improve it yep yeah this information here will actually help us kind of talk to the product team to help them focus their their you know work on where we're developing patterns and and what should we kind of work on next so um this is important and i appreciate you guys sharing so while we're doing that well do you want to kind of walk through the flow and what this looks like yeah let's let's do that so um the flow that you see diagrammed out now is representative it was specifically um initially created for the gcp pattern because that was the capability that got developed first and then the amazon and the azure capabilities were kind of built using the same cookie cutter approach so the same general flow applies even though this specifically calls out some google components so let me uh get my highlighter back here so it all starts with this pattern this is a new pattern that was introduced and there's one for each of the big three cloud providers um called the discovery the kubernetes discovery and schedule creator and a high level what that does and and for those of you who are curious if you update your discovery patterns app you can add like any other pattern you can click into it with pattern designer and see exactly what it's doing you can bring up a pattern log and see you know the steps that it's going through and what's returned back but it basically looks at um it pulls down a list of kubernetes as a service clusters for the particular cloud provider and puts a stub entry consisting of some basic information like the api endpoint the cluster uh the kubernetes uid if that's provided by the by the cloud provider the ip address for the api endpoint and then once that information is collected this pre post script which is also you know you can browse to this within your uh within your instance once you're at the appropriate level of discovery pattern store application this iterates through those and then makes api calls to configure an appropriate discovery schedule which invokes the standard kubernetes serverless pattern and passes it the parameters which were detected in the first step up here most notably things like the um the api end point it populates some new parameters that if you're familiar with the kubernetes pattern you might not have seen until recently such as the cloud provider that the cluster lives on and that's where uh when the discovery actually runs that's how the discovery pattern for kubernetes knows oh this is an aws hosted cluster so i need to use the aws cli to extract the the appropriate credentials and so yeah the steps continue this this is this refers to the the code that was written to actually make the api calls back into the platform to create those discovery schedules and there's code included you know it's it's it's pretty robust if the schedule already exists obviously it won't create a new one if a cluster comes back as absent so cluster was deleted from the cloud then these steps here will take that into account and if it's marked absent the um there's actually a business rule which will take the corresponding discovery schedule and deactivate that discovery schedule and then the rest of the flow just kind of reflects how once these schedules are set up they run independently just like any other discovery schedule they all run in parallel and they use the standard out of the box kubernetes pattern which just has some slight tweaks made to it to accommodate the hooks that are coming from the automated framework versus versus uh being being set up manually and all of those result in a more richly populated cmdb so let's why don't we end the poll and talk about the results and then we'll jump in and have you actually show it in your environment well yep okay all right so you guys should have had the pull results pop up and it's really interesting to see we've got a really great mix between you know kind of size and quantity of clusters and then also a significant mix around where your kubernetes clusters are deployed i'm honestly a little shocked to see so many of them are on-prem right now that's actually really cool uh i i think a lot of the um i think a lot of the cloud providers are doing a really good job uh but it's still nice to kind of have a hand-rolled cluster and then how are they built right build them yourselves is it's pretty small number and i'm guessing that's even a subset looks like that's even a subset of the on-prem folks but there's also a pretty significant number of kind of curated where it's open source tanzu or rancher so that's really really good information and i think what we may do is ask some more questions maybe in the next section to dive into more around that curated support stack and and hear more about which ones of those you guys are using and if you've had any um specific questions or concerns so thanks this is really great information very cool so why don't we go ahead and jump into the deep dive um i'll stop sharing will and you can throw up your instance and kind of walk us through it sounds good all right okay so um i've successfully set up and and exercised both the aws and the gcp option um i'm going to start by showing the aws scenario because it was a little more involved in terms of the aws side of setup to make sure that your discovery the mid server doing the discovery had the permissions required to query the uh the kubernetes cluster what i found with gcp was it just seems to it it seems to just kind of work more seamlessly whereas with the u.s there were a couple additional steps to basically assign a role to the mid server and then map that mid server role to a basically to a user in the eks cluster so kind of start off with that because that was kind of a you know the first time i put it together i was getting all kinds of permission errors and then i had to kind of walk through the just a couple extra steps to get that uh sorted but if i can save somebody else the time of having to kind of uh discover that on their own all the better so what we're looking at here is is my aws account i've got my one of my mid servers here which i've chosen for this task of kubernetes discovery and i've assigned it a role that i just called sn-mid and then when i drill into that role um you know this is a lab so i wasn't you know didn't want to really get into too granular uh permissioning exercise so just gave it view only access for the account which um i use it to discover other cloud assets so it was really you know applicable just so that it could discover the entire account and then so the piece that kind of gives that relevance to my eks cluster is uh using eks cuddle to map that [Music] to map that roll into a kubernetes user so here's one of my i'll just kind of run that against one of my clusters and so this just dumps out the iam identity maps that i established for this test cluster and you can see this second one here it takes the servicenow mid role and maps it to a username named servicenow discovery and so then to kind of round it out i deployed this read-only cluster role which is fairly simplistic it just gives get watch and list against a whole bunch of api groups uh full disclosure i probably didn't need to put all these in but i just kind of tried to uh i went a little overkill to just avoid having to debug a bunch of granular permission issues and then the last piece is a cluster role binding which basically says i'm going to take that read-only role and assign it to whoops let's chat out the actual file so it basically says okay i'm going to take that read-only cluster role and i'm going to assign that to my servicenow discovery user which is the username that i mapped into my role so that that was really that was the documentation kind of says conceptually that's what you need to do but it doesn't really include an example um so i you know i had to do a little trial and error to actually determine yes that's what they actually meant and to get that working i actually did also stick that in a quick community article with the step by step on how to set up aws so you can auto discover kubernetes clusters so um i'll include when we send out the slides i can include a link to that because it's kind of useful sometimes to see a you know to see an example and follow along as opposed to make it up from scratch so that's really all of the extra work that's required if you if you don't have cloud discovery set up at all then you do need to set up you know standard cloud discovery um for aws this this does this methodology does essentially require that you have a mid server running that has the ability to assume or have an uh an aws role so and every once in a while we get a question hey can i discover aws with a mid that's on-prem or somewhere else and and it is possible in general but specific to get this stuff to work it really should be a mid-server that exists somewhere in aws doesn't have to be in the same account necessarily just has to have the ability to um make aws api calls under the auspices of uh you know an ec2 an ec2 permission profile an ec2 role and mike actually alluded to this earlier but just to reiterate you do need the aws cli not the powershell tools if you're running a windows mid it's got to have the aws cli installed because under the covers it's actually making calls to the aws cli to extract the permissions that it needs to talk to your kubernetes cluster in terms of what you need on your instance by default this is not turned on uh there's a property this property right here and this is covered you know extensively in the documentation as well as that knowledge base article that will i'll send the link out to well the link to that is is in the slides this property right here is what turns on that capability so it's got to be set to true in order for your cloud discovery to attempt to generate those automatic kubernetes discovery schedules and so i've got that set to true uh the other thing that's a must set is you've got to define at least one mid server that is going to actually perform the kubernetes discovery because that gets populated when the uh when the pattern goes through and generates all of those kubernetes schedules now there are there's in addition to these kind of baseline properties and this is covered in that knowledge base article as well you can set different properties to point for example different cloud accounts at a different mid server for discovery so for example i took my default one is pointing to my aws mid server but then i defined a second property which is tied directly to my google cloud account um my google cloud account uh project so you use the project id for gcp and so that's basically telling hey to discover google cloud use this uh mid server which exists in the google cloud and so with those properties uh those baseline properties populated and this one especially you know being set to true if we look at our discovery schedules and kick off just the standard aws cloud discovery so i will kick that off and while that's running i'll just go back kind of cooking show style to a prior run to illustrate what the output looks like whoops that's the wrong one again so here's the discovery status line for that new pattern the kubernetes discovery and schedule creator and it's a fairly you know standard pattern log if you're familiar at all with the format that um discovery patterns used for their log output and i can just kind of see yep it's completely happy nothing turned red nothing turned yellow got all the way through and then down at the bottom here we can see where it's running that post processing script which generates the discovery schedules so one thing to call out is the out-of-the-box solution generates a schedule an automatic uh schedule for the kubernetes pattern only and if you look at our documentation where it talks about the manual steps involved with discovering kubernetes it actually tells you to schedule to create two schedules one um which is suggested to run daily that discus that uses the kubernetes pattern and one which runs more often i forget what they kind of recommend initially in the documentation but i think it may be as frequently as like every 10 minutes which uses a pattern called kubernetes event so that pattern is intended to be more lightweight so if you have a really large cluster it doesn't enumerate and walk through all of the objects within the cluster instead it pulls down a list of recent events and then specifically filters out looking for things like pod destroy pod create adding a node removing a node that kind of thing and updates the cmdb based on that activity so um that's not currently scheduled automatically because the concern was that if you've got a lot of clusters that would result in a lot of schedules running on a you know fairly rapid basis and that it could potentially produce you know scaling issues um so i kind of went off and did my own what if exercise just to see what would be involved with taking the existing out of the box pattern just kind of applying a couple tweaks so that it would also generate an event um a schedule using the event pattern and it didn't take it didn't take much i'm i'm planning on kind of putting out an article just saying if you want to do this here's an example of how you could go about it um it's just you really have to be cognizant of what your kubernetes footprint is and how many discovery schedules do you really want to have running at the same time but it did seem like it was a worthwhile exercise that there could be a use case where you know that would be valuable and having those automatically generated would be useful and save somebody a lot of a lot of effort so um it is you know it's it's not there out of the box but if there's interest in doing it it's uh it's not a heavy lift to make that happen okay so our most recent discovery is completed i'll just kind of verify that that's happy and yes if it had failed you'd have a failure over here so let's just go back to the discovery schedule page and so now uh if i pop over here to my account you can see i've got three eks clusters and now i've got three discovery schedules created one for each of those clusters and i'll just pick this one i'll kick it off while i while that's running i will just kind of show how it populated the parameters for the serverless discovery pattern so it basically you know built all of this within using api calls using the information that's discovered and placed in that kubernetes cluster table so it kind of applies a naming algorithm and populates the url for the control plane and then populates a provider flag for the um indicating you know which cloud provider it is using and the credential alias and prometheus url aren't required um there's a property you can set a property either at a cluster level or a global or default value for prometheus url if you've got prometheus running on the cluster the credential alias is completely not required because it just pulls it out dynamically from your cloud provider this is a pretty small cluster so it wouldn't surprise me if this discovery had completed at this point let me just take a look so it applies there's also properties you can use to kind of control the timing by default it basically schedules the initial run for five minutes after five minutes from the current system time when it creates the schedule so it looks like it actually kind of ran it twice or maybe one of them yeah so it actually ran random twice by default it runs it every it schedules it to run every day and so the one thing of interest in the in the logging output is if we look at one of the steps where it actually tries to pull down some information like get namespaces so you can see it called out right here in the logging hey i'm using aws cli to grab a bearer token and so it grabs you know this is part of the output it's not the full token it's truncated um and also by default those tokens are fairly short short-lived um so it's you know it's basically it's using the cloud api to grab the credential it needs and then you can see that it's kind of here's a you know an example of the payload that it's pulling down where it's pulling down the list of namespaces and it just kind of goes goes on down the list hits all the specified api endpoints pulls it in as json parses it kind of the same way that the the kubernetes pattern's been functioning for a while now at this point one of the things i think that's interesting about this is you know one of the standard like best practices is rotating those credentials on a fairly regular basis which was very difficult to do in the any older method for discovery right because we had to have that managed manually within the servicenow platform so having this kind of like triggered and in near real-time basis means that that credential rotation is a lot easier to do and is less likely to impact your visibility into the clusters yeah i mean there really should be no impact as long as all of your you know in aw in the case of aws and it really it it's similar in the other providers you basically have a role assigned to your mid server as long as those permissions are there and you maintain that mapping if you have to with like you do with aws where the cluster sees that role as the corresponding user with read-only access you really don't have to manage credentials at all it's it's really that that's a really big plus i think for anybody who's trying to do this at any kind of a scale [Music] so yeah so here's our kubernetes cluster table we've got a cluster entry now for each of the um for each of the clusters that was discovered and really i mean the the the discovery behavior is identical to what the the you know previous iterations of the patterns um provided the the main you know the the main difference here is the fact that the schedules are not having to be maintained manually anymore actually there's one tweak that i think is is actually quite nice is when you're looking at the clusters the clusters now actually ingest the cluster uid as well which is something that the out of the box pattern didn't do prior to this update yeah you're right that is that is pretty awesome that they're doing that now um they also kind of took the opportunity to give some more meaningful names to the clusters um sometimes the kubernetes cluster naming that it populated in here was non-intuitive and made it hard to kind of because of the way it was truncated sometimes it was hard to actually see which cluster you were talking about by just looking at this list and that is also i think improved with this with this setup any questions okay i'm gonna have another drink of my beer and stop sharing and you can oh sorry well this is rashad from hiv hey how are you doing so great so um i have this uh thought rights we we are mostly on prem at the moment uh would be promoting our on-prem to aks while we are on prem is there a better way to manage the credentials because i'm having to like manually import the credential records create credential alias and tie to the individual schedules that i have because i kind of cascaded them to run one after the other versus all of them being triggered because when it triggers all at once i mean it just picks up and that's another issue i have a case open at support when you trigger a discovery on a cluster also with the hard-coded met on there it picks up whatever mid it wants to pick up from there and it you know when we have a couple organizations in there we don't want it to pick up some or some we want to pick up and also want to use a cluster so that's one thing but then also to manage the credentials is another thing for on-prem is there a way we could do something some innovation to managing the on-prem credentials there there definitely are a few options so um are you managing those on-prem clusters with rancher or is it some rancher and some not rancher so quarter engine okay um so as it ranchers become uh been coming up in more conversations and more and more over the past several weeks and months so mike actually just spun up an instance of rancher in one of the lab environments um so we're kind of actively looking at some options because it came up another customer kind of asked a question about um you know could they consolidate provide a single rancher kind of what you guys were trying to do right create a single rancher level credential which rancher would then kind of proxy over to multiple clusters so you wouldn't have to maintain those multiple credentials so that's something that we're kind of actively looking at we're also i mean as mike alluded to that's one of the reasons that we ask the questions in the poll is to try and bring back um you know feedback to the the business unit that's responsible for this pattern code to say hey we've got x number of customers who are saying they're using rancher and they'd like more robust support for it in the pattern so we're kind of trying to push that in the meantime um the other customer who is running rancher they actually um came up with a pretty ingenious solution it's kind of like you know when life gives you lemons make lemonade kind of thing where they just created an automated import where they just they just store their credentials and their um their cluster urls in a spreadsheet upload the spreadsheet and then from there there's an automated import that goes and populates the requisite records you know which obviously that's not that's exactly what i did i wrote some code i wrote some code we had some 60 to load in so i kind of imported that and then wrote some code to create everything automated it was creating 60 plus or actually 100 at the time when we started and then kind of reduced to 60 but uh yeah doing those manually would have been a bummer and then another question or another issue that i was talking about that i have on the high portal side is about so i have like a mid server that i want the um that's very weird so it's actually it's actually just failing over and trying to use a mid other than the one you're specifying in your schedule yep so initially so initially i thought right so i had like a mid that is tied to a schedule and then i had multiple execution patterns uh tied to the same schedule so all of them triggered parallelly so i thought since they are being triggered parallelly that's why it's trying to pick up whatever met but then when i tied one schedule per one sorry one execution pattern for a schedule and then when i triggered that it still exhibits the same behavior which is very weird and also it doesn't let you select like a cluster if i were to have like a mid server cluster that i want to select and then trigger something so maybe i could do like a schedule with multiple execution patterns and i could use the cluster so it could potentially you know utilize everything that's in that cluster yeah that i mean that definitely doesn't sound right to me um i didn't look at that case i kind of i'll have to put myself on the yeah i'll send you send your name i'll send i will also mention uh i have seen that behavior at another customer so i don't think that is limited specifically to your environment um and that's something that i raised as part of that case as well wait i'll send you the case number just in case both of you guys yeah please do thank you so uh i want to i want to hit uh jason's question really quickly and then i want to make sure we have a chance to talk about there's there's one other thing that we want to kind of squeeze into the schedule today because i think it might be worthwhile so um jason do you want to come off mute and and kind of give us a little bit more detail on on why you don't want to get any of the ephemeral resources so we just did a little engagement with uh aws and kind of with our security teams our cyber security teams and we weren't finding any value in collecting all these ephemeral assets um we turn through sometimes 50 60 000 pods in a day um and there's just too much sharing in there and it's like no one's using them to associate a chain incident problem so it was like why collect them right why waste all these resources collecting them they're just gonna age out no one's ever gonna use them it's filling up the table takes forever to load the forums you know with all the relationships so we decided that we really only want to collect the clusters um there are some tags that are part of the pods that we want to collect because it kind of gives us our app relationships um so we'd still like to take those pods collect them and kind of apply them back up the cluster level so we can service map them still um but but then really uh the only information we want for changes is like the ammo file is like what's important for like our change team right that's what is doing the deployment that's what they're specifying what is changing so really if we can collect that um and uh the image that then is associated with with the pods that's kind of like really the only parts of the environment we want to collect how do you what do you what do you tag the image to without having a pod to associate it to right you're sort of like because generally the way the data model works right is you've got your services right and i could see okay look at the tags on the services that are being ingested and use the services to service map and then not worry about the pods underneath those services right the kubernetes services yeah from my testing yeah we didn't actually have to tag uh the actual images themselves right it'll just be pulled in via the relationship rules right but if you pull that pot out exactly we would kind of lose out yet we lose that middle components now we gotta create a new relationship back down from the cluster to the image right yeah i mean that should be doable i i mean my my initial thought is you could definitely do it i've done it manually and so i'll get it in the pattern now right you know i mean the pattern designer lets you turn steps on and off so you could definitely go into the pattern um so the one thing about playing with patterns is i've learned that there's a lot of value in creating your own copy as opposed to tweaking an out of the box pattern because as soon as you touch an out of the box pattern it doesn't get upgraded when the the store app gets upgraded so it is i mean you know you can certainly just back it off and you know revert it to the out of the box and then it'll upgrade okay but um so just just you know lesson learned that i've had is is it's definitely there's some value to just creating your own copy and and then tweaking it but you could disable i mean you could disable the step that populates the pod table you might like you i mean you kind of said you have to create you may have to create a new relationship because if it's relying on like um like container image goes to container and container goes to pod and then pod goes to um cluster oh yeah it goes directly or maybe you know or maybe node so you just have to see like if that relationship breaks you might have to add a step that says create a relationship between the container to the whatever the node or the cluster yeah and i've been working on i guess my experience with the with the pattern designer itself i don't have the most experience there so i mean i'm working through it i've got to dedupe all my annotations from the pods that i want to collect um now i'm just trying to you know get those to apply on the key value table so i can apply them at the cluster level and then work on creating relationships it's just you know you know how it goes when you're learning an iterative process yep yep it's gonna take a lot of like wax at it until you get it figured out quite right uh yes exactly yeah yeah that's good feedback though because you know like you said with with a rate of change that high the there's like pretty much negligible value in trying to capture all those pods in a table right yep and if they need to do any research they're going to go to the cloud native tools right they're going like divi cloud and stuff like that right where it's got all the really details right and it's pretty much real time for them if they want to get the those pot level details yeah or maybe even to splunk if we're investing in there or something yeah it's just servicenots are the place for them although with the new with the new directions that mike's going to kind of talk about it's early days but that's one of the challenges that they're looking to capture is make it more responsive and give you that like real-time view in the cmdb yeah so um i'm going to run through this really quickly because i i want to make sure we have time for more questions but also i want to get kind of feedback and thoughts on it so um the the pattern team um is looking at how do we redesign event based discovery right so so the issue is right now we have to run those event driven schedules and it's hitting the api on a semi-regular basis and in addition to that um we i think i jumped i got my slides out or that's okay in addition to that we have the potential to sort of not only overload a mid service trying to ingest that data but also if we have a massive cluster that's got a lot of rates of change we can overload the cluster itself so the idea is um the idea is to actually write a kubernetes informer that has actually runs as a background process that then pushes the updates using watch capabilities up into um the mid server right and so as we start thinking about that there's a couple of really good things about that one it it caches the existing state of everything and then it has a um it basically is push based from the api itself so full state comes in initially and then it's synced periodically using the kubernetes api that does the push and that's then fetched using what's called the watch api and so on the on the on the htc agent the full state is sort of maintained in memory and then we can actually take that information and then push it up into the cmdb in order to do those updates in a much faster much more real-time basis but also in a way that's you know less heavily loading on on both the cluster and on the servicenow instance so so the the idea initially is we have an uh a mid server that's running at in uh containerized inside the cluster and there's an agent on there and it's got a check in the background that uses the the client go program to do that um informer connection right and then it'll use a watch only against the specific relevant resources right workloads pods services replica sets and nodes notice we're not doing config maps or or any of the other components there and then that will have a kind of a once a minute batch between the kubernetes api and that informer process that will just send hey here's the updates hey here's the updates so it'll be much more lightweight updates um the other thought process is we could use the acc agent itself with inside of the cluster so that we don't need to use kubernetes credentials to do that and so a couple of things one um the team is actually looking for a customer who would be willing to be a design partner with us on this and actually do some testing on this deploy it in an environment that has a fairly high rate of change that we would feel comfortable ingesting data from and and help to really polish this up so that it can be ready and as soon as humanly possible so one we wanted to get thoughts on does this make more sense and two um you know feel free to reach out raise your hand send us an email if your organization would be interested in being a design partner and working together on this so thoughts questions or concerns okay that being said we've got five minutes left in our normally scheduled time we always run a little over just kind of taking questions and and shooting the breeze um so at that point everyone should have the ability to unmute themselves if they want um and we can oh hang on a second here we can go back to um just chatting and talking about um what's going on in your environment and while we're ruminating on that i'm gonna stick up our exit poll uh folks could provide us with some closing feedback at their convenience that would be appreciated um i have yet to see if there's any way to include like a free-form text field in a zoom poll uh if anybody knows how to do that feel free to let me know but in the meantime um our emails are on the uh on the slide deck you'll see if you got an invite then mike that came from mike's email so if you have suggestions about you know topics to cover um or just you know questions to address it doesn't have to be like a full-blown topic just anything um you know kind of in this overall area we're focused on kubernetes but really we're all about all the cloud native stuff um we're doing a lot with open telemetry now that we've acquired light step um cloud functions lambda azure functions that kind of stuff really you know we're all about all the cloud native stuff so anything in that arena's fair game um you know feel free to drop us a line and you know we're we're trying to make this useful for you folks and so if you have feedback on how to better do that we we really welcome it yeah i'm just going to get some quick feedback i think this is a really good step in the right direction i know previously it's been a couple years since i set up kubernetes discovery against our aws cloud and um you know some of the kubernetes upgrades over the time broke it when they started adding more metadata into the response bodies and um you know just setting up and managing all the cluster information we had to work with their team to provide uh api access into our platform and they were sending the keys in and updating the keys in our credentials and um so that was interesting this is a really good step in the right direction to do this automatically one quick question i was just going to ask is uh you know from the discovery perspective does it care whether or not it's using eks or um you know a hosted kubernetes in you know environment within uh you know just on ec2 and whatnot yeah it's it's specifically hitting the eks um api endpoint so if you've got like just ec2 instances that you've spun up and then you've put kubernetes on those it it won't the that pattern won't pick them up um the standard server pattern will probably pick up your container time but that's an interesting mike do you know off the top of your head if we run like like we're running a horizontal discovery against uh server do we detect hey it's running kubernetes no but but if you're running the docker engine as your it'll pick up yeah it'll pick up the docker engine and it'll pull in the containers but i don't know that it would i don't think that it'll say oh this is a new kubernetes cluster that's an interesting let me just make a note of that well i think you know before we had this capability you know before eks was really i guess formalized right sure yeah you know the kubernetes teams that were hosting in in cloud would open up you know the you know ingress so that you could perform those discoveries appropriately so that's i'm wondering if it something in these new capabilities could still do that and leverage hey you know let's collect all the uh necessary credentials and off uh for those you know those i guess we'll call them legacy by now right right and i think the key thing there is um we you could potentially tweak the pattern to say hey in my network scan i found these ips that look like kubernetes api endpoints and um and then kind of tweak another process to try and go get credentials for those but it would be effectively sort of rolling your own yeah and i don't know how many orgs are scanning their ips for exposed uh endpoint you know um ips for cloud but yeah almost nobody's even doing the horizontal scans in cloud anymore right exactly which actually if you haven't looked at it go look at the aws service graph connector yeah we now have the ability to ingest data from systems management right so if you're running an ssm agents on your ec2s it's a big step forward for us yep that has to be enabled i believe yep yep all right uh thanks for the a things for the questions be thanks for the responses to the polls we we super greatly appreciate it um and and we're over time so we'll hang out and if you guys have questions feel free to chime in and hang out but we also want to be cognizant of your time if you've got other stuff you need to go and do please feel free run on and hit and take care of that and um we will this recording will be up on youtube and you'll get a follow-up email with the youtube link and it'll also have we basically have a playlist of all these sessions so if you are new to the series and want to look back in time and see what we've talked about up until now that'll be easy to do via that playlist