hello and welcome to part four of building the cmdb integration my name is Nick Ryan and today we're going to create the data sources for our integration if we look back at this ingestion pipeline quickly what we're going to actually build today are these first two components down here we're going to create some integration Hub actions which will feed into our data source and once we have all that and the data is is easily retrievable from the third party system then that will feed into the integration of ETL tool which we will cover in the next episode so let's see what it's about in terms of building out the data sources so we need to understand what options are available to us for data sources and then as we start to look at um the integration Hub capabilities we want to understand what is different between a rest action in a data stream and then we'll actually get to creating the data sources so first what are our options uh and we'll see this when we start building it out but when you we open up a data source we can choose jdbc calls um just uploading some files uh we're going to mainly be using actually the custom script option but we will talk about the rest of data stream action uh options as well so as Point number two says there and we'll see this uh in more depth there's a lot of similarity between the rest action and a data stream action the biggest difference is really that a data stream action is one that will continuously uh make calls based on you know pagination capabilities of an API you're using and continue to uh stream that data in in chunks um the rest action really is just going to make a single call out and return whatever response body it gets so it may be subject to size limitations like five megabytes I believe and or things like that another important consideration or design point to understand is that there will be a data source for each query that we make so for different API endpoint that we have to hit on the K Gateway uh to get data about it each of those will be its own data source so we'll have one for the gateway then workspaces and so on and so hopefully at this point too uh we'll be referring back to this Source analysis quite a bit because as we see in this middle column here these are all the different data sources that we're actually going to Define and so uh at the end of this uh this is what we're going to have set up and so I'll probably actually only go through two or three of these you know part of the recording but then we'll show and and talk about all of them that'll be built out at the end so to start let's take a look at what is running quickly as far as like the environment that I'm building against so just working off of my own Mac here um I've got there's some a couple of contain ERS that are running the actual Kong Gateway and then I have a containerized mid server so that I can actually access the Gateway on on my on my Mac and so when we look into the actual instance now here's my mid server that's up and running again this is only required if the service now instance cannot get to um the data source so in this case a service now instance can't talk directly to my Mac behind our firewalls and things like that so I have to have a mid server running to facilitate the communications so now that we've got that I'm actually going to make sure that we switch into our scope because we want to capture uh all of these changes in the scope that we made in the last uh episode so the now Kong Gateway scope is what we're going to be working in switch to that and if you haven't already done so you may want to make a a set of favorites that you'll be working with commonly throughout this I'll bring up my list here in a second but things like the data sources integration Hub ETL uh flow designer and so on so the first thing we're going to actually do is have to define a connection and credential Alias because this is going to specify which Gateway that we are going to develop against so these are some of the out- of the Box ones just's come in here and create new it we'll just give it a name Kong Gateway Alias I'm GNA leave it as an HTTP connection type and it's going to be both a connection and cred eventually at least I'll leave the default retry policy on there as well save that and now we're going to specify a new connection so this connection is actually going to be to that containerized Gateway that's running on my local laptop so here we can sayway in reality you'd probably specify a basic off credential um my this admin API that's running right now on this system doesn't require a credential but um most systems will have a credential of some sort so you can easily come in and and create a credential as well so for the connection URL how it works uh when it's running on my laptop is specify it this format so this is basic say this is the admin API Port so if I were to connect to it through like a CLI it would be the same same type of port and I'm gonna say yeah I got to use a mid server I know I've only got one running on this system you could do all kinds of things with specifying what capabilities it needs if you only want to narrow it down to a certain one and but we'll check this domain because I want to make sure that we're capturing this connection alias in our uh song application actually I believe it is well let's save this and double check it I will come back and validate that at some point but let's keep moving on so we've got our connection Alias defined here's the actual Alias name and this is actually in our scope so now at this point what we're going to do and I'll go ahead and open up the the flow designer because this will be the next step but while that's opening up I'm actually going to come into the data sources so we can talk about uh this for a second so you see all of these um are mostly just example ones that come with the system out of the box when we click on a new one won't actually save it at this point in time or we can just leave it open but this is where we start to see those different options uh around what we can do so we talk about jdbc L have an IDC mostly for like um user Integrations the rest integration Hub rest action integration Hub data stream if you pick one of these um it's really going to then give you an option of where to put that in and so after you've defined one you can specify it there U like I say most of ours we're going to do the custom load by script so I'll leave that there and we can come back to it here in a second to get started though we talked about what's the difference between AR rest action and a data stream action so when we click on the actions we're going to get different types of actions that we can uh or different steps that we can do within an action whereas a data stream if we choose that option it's really going to be expecting a rest end points uh and there's preset steps that you can really do in there and you can't really add as many so we're going to start out and let's go ahead and for ours we're going to say let's get call Gateway see it's in this is our scope this application they we can leave these default settings here in reality probably want to give it some some more descriptions to understand um what the usage of it is maybe what the input variables are and so on but let's go ahead and create our first action so here we are got that so what we're going to really want to do and how we're going to identify our Kong Gateway is through that connection Alias that we just created so I'm going to go ahead and say that I want to pass in a connection Alias and that's going to be a reference to connection alas credential there is have to have that so let's make it mandatory so this is where we can get when you click this plus sign it's going to give us a series of options of different steps that we can take um most of these are just on platform interacting with service now data and this script option is very powerful if you haven't used it before but we're going to go ahead and say let's just make a rest call so here I'm GNA say get info route and so if I quickly look over at the documentation for this an information route and so I'm on the Kong site you see it just takes the Slash and here's an example payload which doesn't look like it really has a pagination uh variable in here that would show me like if there's more data so that we will specify our connection alist that we created so here's so what this really allows us to do is as we move between environments we don't have to come in and change these actions we just changed the connection definition on the Alias and so all of the actions that are using this Alias will inherit that that new host if you're actually pointing from say a Dev instance to production instance so very important that we use use that and really what we want to do here is I don't know that I want to actually pick that one because it Mak change what if you've got multiple instances of K gateways so as we'll see when we get to making the data source if I pass it in and then I use this variable then I can uh change that dynamically through the script load that we'll use I'm going to go ahead and leave this resource path blank because it doesn't need one as we saw in the documentation the error evaluation is a is an optional step but can be good for um handling errors differently or maybe normalizing them uh into consistent codes or things like that uh I'm not going to go too in depth on that outside the scope of what we're trying to do but then the last step is we do have to create an output so let's say that we want to create uh a response this response is actually going to be um a big Json blob so let's go ahead and tell it to be B on then if we exit out of this what we're really going to do is just map the entire response body into that so what's going to happen is it's going to make that rest call to that endpoint and then whatever the response body body back is provide that output now as you see here you can provide back the status codes or error codes or things like that so that can be useful outputs as well again I'm not going to go into depth too much on that but let's go ahead and save this and once it's saved you can see there's a test button here very useful then because this was an input variable that we said had to be there we have to provide our Alias I want the call Gateway Alias and then let's run the test then you can check what this is really going to do is let us know that hey we're actually connecting to the source and we are getting data back so it says code zero is a success click on the response body we see the whole payload that came back so great we know that we are connecting to our data source that's great close all that close this and at this point we can go ahead and say let's publish it publishing means that it can actually be used um by the platform so once you've published it now we can actually call it from our data source start to start making some some Headway there so we've got that I'm going to go ahead and maybe make one more um action while we're in here and so uh a quick and easy way to start doing that is actually to if you've got one that you know is going to be very similar uh you can say all right well let's come in here and copy this action so all of our actions are actually going to be very very similar uh just a little bit different on the endpoint so here I'm going to say what the second one we're going to do is we are going to get Kong workspace and once I copied that then I don't have to go through some of the steps like if I'm always going to pass in that connection Alias variable to start out with I don't have to continue to uh do that over and over so here it is and good to go in this one I actually know that I'm going to need one more variable if I come and look at the documentation for the workspaces what you'll see is uh that's actually the post but if I go down to listing workspaces it's a get call against this endpoint excuse me and here's some sample output if I go to the bottom one thing I noticed is that there's a next this next actually means that this API May paginate so I I need to account for that and so in this particular case I know that I'm going to want to pass in uh an offset variable because that's going to be what comes in uh in that we're going to have to change this to actually be get workspaces as we saw and we know that that endpoint is actually from the documentation it was Flash workspaces so if I put slash workspaces and then what I actually want to do after that is provide this offset value because and we'll see come back and see what that really means in a sec again not going to touch much in the way of error evaluations and again our response body is really what we want there so we can save this we can test it and don't have to give it the offset but we'll see how that plays out and let's check this out all right success so here there's only one in uh in my test setup and it's the default one so that's correct there's a bunch of information about it but that's the one if we look in here at all our workspaces I only have this one called default close that that let's go ahead and publish this as well while that's publishing I'm going to actually go over to back to the data sources and let's start building our first one so the first one we saw was that it was a k Gateway and I'll just call this the thing and it automatically populates the Imports at table name and so here there's a number of uh variables that we'll explore as we get into this I'll make this a full screen and for the sake of not the one thing that we actually will need um from this is going to be our con Connection in credential Alias that we created we're going to need the CIS ID of that guy oh I'm going to come back in and grab that quickly copy the CIS ID let's come back in here this so I'm going to build this out a little bit slowly the first time and then we'll speed along after this um always good to put in and it's really good if you would put all this into a script include to encapsulate it and then make calls to that from here but for the sake of speed I'm going to um not I'm going to skip the script include step so our Alias ID is that one I just copied then we can basically well the first step we want to do is get the Glide record for that and and if we actually get one of those start to build out so what we're going to do at this point is we've queried for that connection Alias record because that's it's a reference remember in our flow designer action we made it a reference to that record which means we need to pass the entire record in um there are some certainly some other ways of doing that and I know there's obviously some other ways of making these queries with get value and and things along those lines so code at your own preference I wouldn't say this is the best practice on coding techniques but see how this works we're actually going to now build the inputs for that rest action that we created so I'm going to say inputs and then our first one was called connection Alias so if I go back into floated deser I open back up this one and we look here so connection Alias this was the variable that I'm trying to populate now so connection Alias equals our Glide record that we just queried for and that was the only one that we had one thing that you can also do in flow designer if you don't already know this is actually get a code snippet so you can um come into this little related list over here and say create code snippet what it'll do is it'll show you how you can actually call this from from a connection so here like we said right the the connection inputs and then it's really this line that's kind of long and confusing but I'll I'll go ahead and get these two lines from it and back in here a nice thing we can clean that up there click that so now at this point what it's done is it's called our G Kong Gateway action it's passed in the inputs and off we go got some outputs here and we can start to start to use this at this point um if we want to there's some other things you can do with this import set table um record and some of the the other things in here I'm not going to go crazy in depth on them but if you want to preset the column names there is like a import record um add column capability so so if we start to type import record in here uh you'll see that there's a few different things in here um that you can that you can do and so I would explore some documentation on that if you want again I'm not GNA go too crazy on these in this video I'm going to just initialize the import record here and then we get can start to populate it with things that we know are in the payload so uh if you just start to uh type these in we'll see how these these names of this object are going to translate into the import set they'll usually be actually un prefixed with the U underscore um one stuff I forgot to do actually up here is to code our actual response body because remember it's just the entire giant response body we want to gets so again if we go back to um our flow designer message or our flow designer here we see that uh this is actually called response so it's important that it matches this in terms of like capitalization and all those things needs to be the exact same name and so I'm going to decode that so now I can use this object and start to just pull out values that I know exist in the payload which again you can see either in the the documentation or um you know if you've played around with it in hostman or so on so in the documentation you know there's like a host name node ID plugins versions so on so you can look at it that way but sometimes it's not a comprehensive list there so probably best to to play around with it and see what what dating you need and it also does come back to are mapping hongway mapping here again these are the fields that I know that I want to get and so you can also should be able to use that to specify these values for the sake of time I'm going to go ahead here and I have a lot of this already out that up go and so you see there's a number of these that that I'm getting out of that payload the values the connection ID Alias that we want and then basically we insert into uh this overall parameter uh the value that we want double checking all this because looks like something maybe is a right go so should be good there one save it so now we have our first data source in our application it's making a call to the action and we're good to go you can actually test it here by running a test load of some records or all the records um I'll save that for another day but uh let's work on one more of these just to give you another example in um even beyond that data sources we can actually start there again and just take and copy all of that over uh I will expedite some of the typing on this next one because not terribly fun to watch but I will talk through code on this other one that I'm going to do so the second we made that other one that was for the workspaces and so let's go ahead and create that one here as well see e-c workspaces and again we're going to use Lo by Script and so this is really maybe a better example of how the custom load by Script can be a little bit more useful so there's a lot in this one if we actually um again more time different one what this is doing same thing so I'm going through and getting the Glide record because that was one of our inputs obviously and then so was the offset value was another one but you'll notice what's different in one of the big differences in this is that I have this do Loop and so this do Loop will exist in all the remaining data sources um because all of the remaining API calls we're going to make support that pagination and have this concept of an offset and so you'll see that this is actually going to be same here this is check out that one make sure we get the same name on that create the code snippet so we're going to pass those two variables in do the same type of thing got to decode the entire response body but here we're looking for that offset so if the next value uh again workspaces documentation all listing workspaces if next is null then there's no more data to get but if it's not null it will have this offset value in in a basically a URL and so we want to parse that offset value uh and what's after it because what's after it is it's a little looks like a CIS ID effectively so U so we set up this Rex up here to basically look for the offset value in that string we're saying if if the next value isn't null then go ahead and match it against that and and create this offset value to pass in uh and then the other part to know about this if we look at the workspaces is that it's actually an array so there's this uh data array that we then need to scroll through so what what we have after we've gotten the response body is that we need then need to go through that data array and and bring in each record in the array and so that's why we have this little for Loop to handle going through the array here there's not much that we're going to get except for a name and an ID and again we want to pass in this connection Alias so that we know which Gateway we're we're talking to so at this point that's really it we're uh We've created a a data source for the workspaces as well now and so this this same type of pattern is going to happen over and over so at this point I'll actually going to pause the recording and we'll come back once I've created all the the remaining ones okay so we're back and now we have all of our data sources defined and each of these eight data sources has a corresponding um the corresponding Low designer Act action associated with it so I'm going to get in here very quickly and actually show one Nuance about these um that from we did the Gateway and the workspace together but now the rest of these actually have a little bit of a Twist in that they are actually going to provide an option if you don't want to import a certain workspace so what we're actually going to do is introduce a new Loop into this where we're going to um Loop through all the workspaces that we loaded in that other step and then follow the same pattern after that and call it but what this uh allows us to do is say in this load balancer request we can pass in the workspaces only the workspaces that we actually want to to pull back so things to consider when you're interacting with apis like in the case of Kong this actually isn't required I could just query directly against the upstreams and it give me all of them on the Gateway but if you want to um control things a little bit more then look for some of these additional parameters to work with and so the last thing I'll show before we wrap this up is we've got all of our data sources loaded probably a good idea if we actually save them somewhere so we briefly touched on and the third episode how to create this app and the fact that we can link it to source code so now once we've saved some of our changes this is really a valuable part of doing it and working within a scoped application is this step once app engine Studio run opens up here we'll see here's our s Gateway app and once this summary page opens you'll see for Source control I want to commit the changes so I've already Associated it with my GitHub repo and now I just want to commit the changes so come in here really review them all but let's go ahead and continue comment first commit with data sources defined we will commit all those at this point we've gone through created our data sources and our flow designer actions so we should be having the ability to get our data flowing into the system and in the next episode we will uh actually use that data to start building our transforms and our Integrations in the integration H ETL so until then good luck with building your data sources