foreign immigration Hub and data streams so just a high level agenda here so what are data streams and why should you use them the components of a data stream and then what everyone wants to see is the demo of it in practice that's the interesting part all right so why should you use them so as we'll see uh in a minute it packages Integrations into a nice container there's a number of features in this that are probably not going to be used a lot but it really it puts it all into one so you can you can have it well organized it supports paging and large responses that's really the paging I would say is the primary reason you would use data streams if you're not using paging it's probably a little bit of an Overkill it's an action and flow designer you get kind of like all the built-in logging and error handling that you would get with you know any of the I mean the other flow components so that that's all kind of built in for you how do you get it so it's in Immigration Hub Enterprise pack so one of you know the kind of the highest level there you can activate this on a pvi if you go into the developer site and you know activate those digital plugins so you must have that otherwise you're you're not gonna you're not gonna see it all right how do you get to it so you're basically gonna open Flow designer you're gonna click new the new button in the top right corner as I'm sure a lot of us are familiar with and then you'll have this new Option of data stream so you'll click that and we'll look at the components of the data Stream So basically from the creation and kind of how it tabletizes this uh this special kind of action so you're going to get um once you hit create the new action you're going to get your standard action property screen here so your name your application you know if you're if you're developing you know their scope and then you know we're successful from so this is this is not special to data stream actions this is special though when you create a data stream action you get this outline of things and we'll talk through through each of these because there's also options within each but it breaks them into these these different sections so you have inputs which is a standard thing with with actions and you know Flows In general action pre-processing where you you can perform certain actions before you make any requests at all the requests which we'll look at has this is where your actual request to your system is happening then you're parsing up your response and then your outputs we'll talk through each of these in more detail the first one actions not not a whole lot special for the sake of the demo um I create an input called table this is just going to take a table name and that'll allow you when you get to actually configuring the import that you can say oh I want it to be this table the pre-processing is this as it says here in the help it's going to be executed only once per options so there's probably not a lot of cases where you have to use this where you have to necessarily do something before you're making any kind of call but I can't give one real world example where I had to use this so one of the immigrations I was doing was with a system that was not using oauth not using basic authors using an API key but the API key was actually a session token that expired like every two hours so at the beginning of of the process here I use this to check on whether or not the token was valid and then if not refresh it so that was that was something that made sense to do at the beginning of the call there's also another spot this can be done but that's that's one kind of real world example where you know if you're dealing with kind of a unique authentication uh condition you could you could put that in the action pre-processing so the next the next is uh your requests set up here so if you look on the on the requests action here you have a few options here so you have enable patient which we're gonna we're gonna talk about that's one of the main reasons they're going to use uh a data stream and then uh run a script before each request this is a lot like your action pre-processing that we just talked about except in this case that would happen before everyone every request so again with the authentication example it might if your time with further request processing takes you know you know a long time or or you need to you're concerned about you know Crossing some kind of race condition where your token expires maybe you want to check that before every single request you make so this is a another spot where you could run that pre-processing much like the on the request unusual request instead of at the beginning so again probably not a something that's going to get used a lot but it goes it was a nice option to have this is where things get interested in with the paginations there's a few really important um Concepts here so the first one is there's this out of box initial uh patination variable that is used to communicate whether or not there are additional pages to be processed so it's called get the next page it's such a false and it's going to get it from the script you always have to set it to true if you want to process additional Pages if you just leave this it's going to hit that first page and then say you know oh my value is false so basically how you do that um is you're going to do some calculation of your script here in this case use the table API August servicenow what I what we're doing here is you you're we're grabbing from the header what the total account of of items is and if the offset is not the same is not equal to the total count it's going to to set the offset to the next one and then set the get next page to true so this is gonna on every call is going to look at the data and then you have to tell it whether or not you're going to process the next page and then usually you're gonna whatever system you're requesting to you're going to have to say okay get me this offset or get me this next skip this number of Records um it's really going to depend on the API that you're that you're going against what you'll want to look at too is the page response for what data comes back it's usually going to return the it's going to return the headers and some other data so this is an example of what it looks like from the table API so the response headers is going to return you know the server you know these different header values and the the one that I used in this case was the total count so against the uh the table API for instance I have 67 records so I know that I need to set up my paging spill to handle that what I did for for the sake of this demo to prove this out is that it's only going to return one record at a time so this should go through 67 Pages basically of Records a record length of one there are there are other Advanced pagulation options here there were some challenges I found with this where what you can do is you can say Okay I want to look at the at I want to set up password variable based on the response body and you can extract that value that you can either use Json or XML here depending on what your what your system is returning but this uses Json path and in this in this case we're saying the response body from servicenow returns an array of results I'm getting the tenth one and the number and that this this is just an example so one of the things I was not able to find that I'm still looking for is what capabilities Json of Json path and what what implementation servicenow is using there's a link in this here to a Smart bear page that has what the Syntax for Json path is some implementations of it allow you to say you know get account of Records so you can say it I think in the Json case it's you know you could say like result dot um dot length or size I forget which one it is but in the implementation that servicenow is using it doesn't appear to support that so it can get it can get a little tricky about um how if you're dealing with it from the response body about how to get that if there are additional records available but this is this is one of the ways to do it the preferred way would be to get it and and the way that it seems to handle the best is is getting it from the page response and the and the headers hopefully there's a header value in there that you can use the other uh this was uh we kind of talked about this uh a bit where this is the other kind of script step you can use before each call again you could use this for potentially authentication and then you get into your rest up here so this is this is very similar to what you would use you know in a in a flow typically and we'll look at how how the parameters are set here in the in the actual demo parsing this is another thing you have to set up uh really you don't get a whole lot of options here um it's pretty much going to be your Json XML splitter and then you're gonna you're gonna parse it with a with a script here and what that looks like in the splugger step is like this so in in my case here we're dealing with Json the item path um this is another kind of Json query that you're gonna tell you're gonna tell it basically what's the array of items I'm going to look at so in the case of servicenow it's going to be a result and this is the syntax you use to tell it like this is where to look for for my right the script parser is this is where you actually go through the rows that you get back and you and you pull out the demo so you have to configure this on the outputs together I'll show you what the outputs look like this is where it it's a it's a little bit of one of those servicenow magic things where you're like okay where does like Target object come from and and how does that connect to my output um and I'll explain how that looks because that it can be a little confusing at least it was to me so I'm going to parse that Source item and then for each trial you want to pull out from your response you're gonna you're gonna set that to an output variable so in this case we you know outputs our Target object which I'll show you where that maps to and then the actual property so number is you're going to be your item and your number or you know in this case we also have a short subscription which is item Choice description so this is where you're going to map those uh items when you first open the script parser step you'll be able to see liked examples for uh Json and XML that servicenow has has provided next so this is the outputs this is where you actually have to set this up before the uh the square person will will actually work you technically can set them up separately but this is a requirement for it to actually work so you basically set up a an object at the top it doesn't matter what it's called which is the part that's a little confusing so you can call this whatever you want as long as it's an object and that's what Target object is going is going to be and then under that you would create each of your properties so in this case I just have a couple of strings number and short description one of the things to keep in mind too is that you may need to set the the max length as I'm sure a lot of people have experienced right if you if you let it set the default length of of things like with strings it'll be 40 characters right a short description you know is 100 characters right so it's gonna it's gonna chop off those values so you're going to want to for those longer string Fields you're going to want to come in here and configure and configure the max length those are all the steps to actually set it up and then how to use it one of the ways the primary ways to use it is to set up a data source so uh I've highlighted some of the important attributes here but basically you're going to name it you're going to give it a import set label and table and then the important parts are you're going to have this option here that says uh data stream type integration Hub and then your data stream action one thing that of note is that it won't show up until you publish that action uh so you gotta make sure you publish it and then you can see here under our data stream uh action inputs it shows that that our input that we said was required a table is is here and I have it set to incident just like any import you can use transforms are the robust Transformer to map that data into a system in this case I just have a my source table on the demo actually and then my target table of uh ticket lists this is our the demo action I created so we'll we'll kind of look through each of these and then we'll do we'll do a run um to see to see what it looks like so we have our our input here which uh we will go too much into it's just our our table our script step our pre-processing which I left empty you can just turn it off it won't even show up then our request so again you have these different options I have the before turned off and then the pagination and then in the in the case of service now um this was on the screenshot as well you know we're using the response header here adding one to the record since I told you told it to give me a page size of one uh to prove this out uh rest up here you usually not gonna obviously do this against the instance itself but uh rather a external instance put forward the the purposes of this um it's just using a connection Alias that's pointing back to the instance itself with a base URL on the table um we can see here that our resource path is going to be the the table I have a limit of one and then here it's Gathering the offset from the output of that pagination um I added an additional transformation here of two fixed uh one of the things I found was that even though I told it to be an integer it was adding decimal points of dot zero to it which the offset will will work if you have it not an actual integer but yeah it's just coming from our pagination step here you don't actually have to do anything with the uh get next page variable you just have to set it and then it knows okay I have to keep execute doing this until until you tell me it's there's no more records our sweater step uh yeah so our parsing here right there's really just these options um available for me so the splitter is just gonna pick your Source format and then um your item path which was that Json path syntax and then the servicenow case that's going to just return a result array and then in our our script parser step we have we have this you know these examples which you can you can take out their comments but essentially like we just looked at you know you're gonna parse you're gonna parse that input this is a kind of a little bit of magic where it's where it's there you have your your input source item that's there and then your outputs Target object is where you're going to set your your various properties if you're used to dealing with outputs and flows under it's a little weird to not like have to you know pick a value and map it in here it's coming from the script parser step where you're mapping to number in short description so you just have to make sure that you have those those outputs there and then again that top that top action output object the name doesn't really matter it can be whatever you want it's going to use the target object to demonstrate where you see that advanced option so if you if you click over here you this is where you can set the latest all right so let's run a test and kind of see what this logs out for us it's uh going through and processing each of each of those pages that's why it kind of took a while right because it's like it's doing it's looping through instead of grabbing all the records one of the things when you do it when you do a test it's only going to pull it's only going to pull um 20 pages so that's just something to keep in mind when you run it from the the data source it will it will obviously pull all the records but you can see here that it it mapped my output items I mean this is it's a little funny though it's output space items but that that does this on a box but you see each of the items that pulled back my number and my showing description there's a lot of uh details about how the different variables are being set what the offset is uh at that particular point you know what your response body looks like so it does a fair um amount of logging uh which is uh configurable obviously but when you're developing it's really good to see okay what setting for the packagation and what kind of data am I getting back is it mapping correctly so there's a fair amount of uh data there so on onto the data source so I already have this configured one one thing to note here is that just like if you're doing like an Excel import or something like that you don't you just have to stack up the table name and it will actually create on your import set table it'll create the fields for you so you don't have to go through and set up that import set table you just you just go unload it um and it'll it'll have the fields there for you so if I load the records here and obviously you can schedule like you would schedule this to run right as a as a scheduled import but it loaded these records in um and then uh you know you can run your transform as usual and right so it pulled in and pulled in these values from that data source and if you go back and you look at the executions you should be able to see that you know this one ram so it got it got all 67 right and you can you can see the proof of that here that's essentially how this works there were a couple of thing things uh you know I found and when talking to other people to keep in mind so that the patient can be a little limited and complicated my experience was that if the uh the system wasn't returning the you know the source system wasn't returning better with like you know what page am I on like how many records are left it can get a little complicated to have to look at the response bodies to get that data especially with it's seeming that the the version of Json path that's that's provided doesn't have all the functionality one of the other things that I definitely run into and I know that uh um with helping out a couple other people that this has come up like if you've got a weird error make sure you make sure you're actually getting data back because it it like especially from the uh the the data source it won't say you've got no data it will it it throws like some errors that don't necessarily make sense so what you really want to do is is uh if you run into a error on your data source import go look at the execution history for the action and make sure you're getting data back and that and now it's actually working correctly a couple of uh resources here uh there's a uh data stream of actions uh and Pat education doc site um which has some pretty good information uh it is a little light on the hydration end of things so that's really the most complicated part of this and then uh Chuck did a a really good video about about the data stream actions as well uh so I would definitely um check that out for more information but that's that's the end of it um what questions uh if not only do people have were those data streams um do they utilize the retry policies and if so I'm just curious how that would work when you're paginating I haven't configured one of those yet but you yeah you can it's just like a regular just like a uh um okay you have a refund policy here so you can use the the default logger to find your own but but yes yeah that's cool I would just be curious you know if if it's paginating through 20 pages of data and then on page 10 it's like oh there's an error I'm wondering if it retries that same page or if it just goes through if it just stops or something I'm not just curious about that it's not going to start from the beginning it should start from whichever step it was on um since it's in this in this kind of like request um section so this is like it's looping through in the in this case so it should just be that just be that particular one but yeah definitely a real world thing right like you know before the existence of this it definitely written Integrations with page and you're like okay like now I have to handle like what if it fails on this particular request right yeah yeah thanks and then uh really just for me one Integrations in second flow I always have a hard time trying to figure out the best way to approach error handling um whether that's you know if you counter an error do you create an incident do you notify someone like I'm just curious on your thoughts on that yeah I mean I'm curious what other people think but my generally um I like to create an instrument um because the question always is like okay like does it get assigned to the the service now the team does it get assigned to the the person who is supporting immigration so it's really going to depend on like on a situation a customer my customer basis I would say and that's a that's a service now best practice to create an incident off an Arab integration just be careful of all your deduplication like if you get thousands of Errors you don't want to create thousands of uh incidents and make sure you mitigate that but otherwise yeah it's definitely a best practice foreign [Music]