hello and welcome to this bonus segment of building a cmdb integration my name is Nick Ryan and in this video we are going to cover some steps that we didn't see when building out the full CNB integration for Kong gateways but are very important to understand and may come in handy in your own Integrations so let's take a look at some of what we're going to dive into the first thing is not so much anything that you would need to do to enable this but it's good to be aware that the integration dashboard will actually reflect data about your integration that you build without you having to do anything so we'll take a quick look at the dashboard the Kong Services ETL definition that we created had a number of things that are important to understand should you run into the situation where you want to use the same data source to populate multiple CI classes and we'll start to take a look at some of the scripting options available in there as well uh similarly with the Kong routes ETL we'll see when it might be useful to do CI lookups and to use that transform option and there's a multi-input script option as well and there's some things that are a little bit interesting about the relationships that we built in that ETL that I'll point out and then finally we'll look at how to use the CI lookup to populate some reference Fields as part of the ETL process so let's jump in so to start here is the integration dashboard so if you're looking for it it's it should be under the service graph connectors menu and then there's an Integrations dashboard there and so what this actually will show us over here on the right hand side is all the applications so if you have service graph connectors installed they would show up in this list as well but the cmdb application that we just created the now Kong GW application shows here as well so you can filter by that and it'll show us a number of things about how many Imports have run over a certain time period and some useful information about which CI classes are actually getting uh populated during this this time frame and so even though we've only got one calling Gateway we've run that import a number of times and so it's actually not just what's been added but what also has potentially been updated so you see it's been unchanged 64 times so that would be expected in this case so going beyond that let's jump into the next area that's interesting in terms of looking at our Kong Services ETL and going into some of the more nuances around this so the first thing that I'll show you is that in the the case of this mapping I actually wanted to use this same data set to populate the managed API class and the API backend and that's because Kong is a little a bit nuanced in that they don't have a the concept of a higher level API object that many other gateways have so we're going to reuse the back end or what they would call the services in Kong to represent both the back end and the actual API in this case but let's go back and actually look at some of the data that we needed to create in this so when we're looking at the transform section of this you'll see these columns in Gray and all the gray columns are things that we created as part of this the columns in white that are not shaded are the native columns that came from the Imports and so that's what we have to work with and so we want to do some Transformations like um one that there's a a convert to numeric one because everything comes in as a string unless you specify otherwise during the in the data source somewhere so we're going to lot that dot zero off of that and you'll see here it is in this part here but let's start out with the fact that because I'm mapping to two different classes remember the the managed API and the API backend I need to have two different Source native keys um for that and so it's not terribly difficult uh the source native key for the API is the actual service ID itself so we brought in the ID That's Unique in the Kong Gateway and so that's pretty easy we use that for mapping the managed API but for the API backend we need to create a different one so we actually use that same ID and we just concatenated back to it and so if we look at the concatenation feature though what you see is that you don't you have to put in two actual column names and so you can put in more by clicking the plus button but you have to put in at least two so I want the ID and then I had to create a column just with the value of back it's just a static value so you'll see in the transforms set a fixed value column so that's going to be used a lot throughout these Transformations where we have to do things like concatenating a fixed value to that so uh that's important to for making sure you're forming your your Source native Keys um to be truly unique in this case and so let's look at actually a scripting option here so there there's different types of apis we've got rest apis which are the most popular but there's soap there's web sockets there's grpc and so on and so Kong actually will tell us in the protocol value if I go all the way over to the data that came in there's a protocol and so you'll see that there's these different values in this case but it also could be WS WSS grpc um sgr PC there's there's different values that can come across and they represent and can tell us that these are different types of apis and so I want to populate because it's actually a choice list on our table on the manage API table it's a choice list for the type and so I need to match those up so that it populates correctly so here I chose the script operation to do this um I said this is the output column so you see API type is is our output column and then the input is that protocol field that comes from column so uh that input actually comes in the form of an array and so that's why the the scaffolding for this when you if you open it up by default it will have this for Loop and this input value here for you and then you fill out the rest of the code and then this output field needs to be set but in this case I'm just saying all right my default is going to be rest and then I'm going to look and see if the protocol includes WS which would indicate that it's a websocket or if it contains grpc then it's a grpc type of API and so that's just one example of how to use a script we'll see a couple others and go along here and then I'm just going to set that type value in the output so very handy if you have some Advanced logic that you need to implement in some of these transforms and so one other one as we look at at this is the Kong load balancer Source native key so let's take a look at this transform and this is also a script operation same same type of concept I'm going to use the host as the input here and what we're actually going to do is actually query the system now this is something to not get carried away with because you can create some queries that will have an impact on your processing speed so I would say to Quick use queries in these scripts very sparingly and certainly make sure they are as efficient as possible using indexes and so on but in this case I want to find the ID value of the load balancer because I want to know I want to actually make a relationship to a Kong load balancer so I need to get the source native key and for the load balancer in this case and so this script is just going to go query it based on the host value and and get us what we need so let's take a quick look now at some of those the mappings I'm not going to go too much into them again we just set the source native key on the Gateway um and similarly on the load balancer that that Source native key that we just looked up in the script all I'm doing is setting that so that I can find the appropriate one when I get to building relationships and then in these mappings again we have the source native key was the ID which is important to understand being different from the back end and so we use an ID there and then populate in the fields as we as we need for this on the back end here we're using a different Source native key because if I were to use the same one as the managed API then those two would actually be in Conflict they would be overwriting each other which is not good at all and there's a few more things that we're filling out with the back end but that's pretty much it and then the last thing on this ETL that we'll take a look at is the relationships so this is where we're building out and again not all of these are dependent um this first one is you can see it actually says dependent relationship here from the gateway to the managed API and then our managed API to the back end is a suggested relationship and so it automatically populates this uses and used by a relationship and similarly where it is applicable where we're actually the back end is pointing to a load balancer or what's called an upstream in Kong we're making that relationship as well so that's it on the services one let's take a look at some of the things that we did in the routes ETO that are a little bit unique compared to what we've done so far um you can go into this transform section here in Step number two I'm going to go ahead and jump right into where we're going to do the mappings here um on all of these also we're just setting the source native key for the Gateway similarly with the managed API all we're doing here is setting the this Source native key we're not actually doing any updates to it and similarly with the back end so in the services ETO we populated manage API backend but now in this routes one we're actually just wanting to populate front end but we need to make relationships to these other ones so that's why they're here as well and all we do is populate the uh the source native key so we can make sure we find those appropriately for relationship purposes but we'll come back to this but we can always get to that transforming the data in the same spot as the mapping so if you need to add some more you know the toggle back and forth but we'll see here that we actually have a number of fields that we've also created so the first one this API base URL so in Kong the routes don't actually come with the full base URL they have a lot of paths uh in front of them and other data points but we want to get the base URL from um from the API record and so what we're doing here is actually a CI lookup on this and there's some important things to understand about doing this lookup because it's actually going against not directly to the cmdb tables it's actually looking at the CIS objects Source record and finding the data from that which is why the inputs for it are the discovery source The Source native key and the field that you want if we jump over here real quick and we look at this object source yeah and we'll see a number of these what you see is that there's this Source value right and the feed is actually where it's coming from like our data source in this case and then this ID is our source native key and then you see it's going to point to the actual CI record and so what this function is really doing is looking at that sys object Source table we have to give it that Kong Gateway source so I actually set a static value here for now dash Kong GW because that's our source as you see here in this this field over here and then the source native key depending on what it is like in this case we want to look up the service that this route is associated with uh because that's going to give us the managed API Source native key which actually it's what the field is that we want to get from the managed API is the base URL so I actually had to set that value as well because this as you see here there's this column that I made that's just again one of these one of these set fixed value columns but the input here is not like a free form field it's it's actually you have to pick one of the columns that's in this data set so that's why I created a column called base URL and just gave it a value of Base URL there so that I could use it in this lookup and we're going to use this CI lookup and some some other places as well but the result of it is that it went and found the managed API value and gave me the base URL attribute from that CI so that's what I have here and what I can use for that actually what I'm going to use with that is to create the full URL and so this is a little bit different in that now we use we looked at the script operation before but there's actually a multi-input script option as well so if you don't need if you need more than just the one field or column to work with then you can have multiple here so in this case I'm using that base URL that I went and got as well as the path for this route and then you could add more inputs if you want to but those are the two that I need for this for creating this full URL and really all that I'm doing is again it's it's very similar they come into this is kind of a batch option and you see that the you you can actually reference by the column name that you're using there and and that's how you get to those each specific input um from that standpoint and so I'm I'm doing some stuff about stripping off the any trailing slashes if it happens to have one because the path value actually comes with a slash at the beginning so that's what's happening there and stripping that off of the base and then I'm just adding the base and the path value here and giving it that to the output so that's how you can use multi-input Scripts are useful as well and the last thing to really see about this ETL function is again on the relationships and building those out and so we set the source native key uh let's go back in here we said snores native key on Gateway manage API and this back end and really we just populated front end so we have all the things set that we need to in this mapping step in terms of finding the right CIS and then in the relationship steps again it's pretty easy to just add relationship to choose your um choose your parents choose the child and if it's dependent like this one is it will pull in the dependent relationship automatically if it's suggested it actually will also pull in the suggested relationship type but you can see here you can actually change that so in this case the managed API uses the front end and in the case of making the connection between the front end and the back end because I want to know that hey this front end is also communicating with this back end there is a suggested relationship between those two classes as well and so it automatically populated that when we pulled that in so that is it on the routes ETL in terms of things I wanted to cover and last but not least is on this plugins ETL which the plugins is actually going to a related table and so if we look at our mapping you see there's the Gateway but the Gateway has a bunch of plug-ins and so it's actually a not it's not a cmdb table but in this class manager it is a related item in term when you're setting up the identification rules and you can set up related entries these plugins are actually related entries for this specific Gateway and so here on the Gateway again we're just setting the source native key but if we look at the mappings for the plugins we go to transforms this is again some things that are are very useful to understand so again I had to set certain static values here as you see this is just setting the static value of back because we're going to use it as the suffix and for our back end Source native key so if you recall we did this similarly and when we were setting up the in the services ETL and we had to set two different Source native Keys we appended or you know we appended the back word uh to that so we're looking for that and then from that actually we're doing our CI lookup again similar to what we did before and I set the data source so here's our data source value that I had set I want to use this back-end Source native key and I want to get the sysid field so again I set a column that's a sys ID because that's the attribute that I want returned and so what we're going to do with this is actually create references as we as we map our data in and so we did that the consumer also we did the same thing here did a look up on this so using so the the raw data actually comes back and tells us all right there's an ID of the plug-in if it's associated with the route it gives us that ID if it's associated with the service gives us that ID and if it's associated with a consumer it gives us that ID so the raw data is what we're using to as the source native keys and so we specify that so give me the sysid so again if it if it were populated then it gives us returns our society and then we do the same for the front end as well if we look back at our mapping how that plays out is that we then will say um these references to CI classes go ahead and populate with this transform column and pass in the sys ID for the front end because it's a reference field same with back end and in the consumer column even though it's it is a reference as well it's not a tnp class and so we're going to pass in the Sid of that as well and so with that we've seen a fair number of ways to use the different transforms available subscript options and what those can do for us excuse me and hopefully that helps in some places where you need to do something that's not very straightforward in terms of massaging the data and mapping and references and relationships so good luck in building your cmdb integration and hope these videos have helped foreign