[Music] hello everyone today i'm going to demo the end-to-end flow for ihetl nested payload feature first let me give a quick introduction about this new feature before nested payload feature the only way for a customer to upload their raw data into etl app is through a flat import set table but it is very natural that from a modern system customers raw data is nested data such as json for example a computer that has some storage disk and the software installed this kind of structure cannot be represented by one single flat table currently customers have to do some script operations to transform this kind of nested data to several flat table data this script transform process is difficult time consuming error prone and not performance efficient but with this new ihetl nest payload enhancement ingesting nested payload such as json becomes a breeze now i will demo how to create an epl map using a nested payload the first step is to upload the customer raw data through platform data source previously customer has to put their raw data into a flat data format like csv file this is the old format like the csv format they have to use basically it's just row by row after this flat data source is loaded each column of the raw data corresponds to a column in the import set table so basically if they load this into their data source and it's going to be loaded by the platform transform and this is the import set table they're going to have and we we can see that the columns correspond to the csv file as we can see here the computer name and we have computer name here in the csv file however with nested payload feature customer can put all the nested data into a single nested structure like json or xml let me give an example this json object comes from the gmf customer's actual computer information i pulled out this json from out of box jmf computer data source as we can see here this computer has nested storage disk which is here this is the storage device which is the disk and also we have the nested software installness here each nested level has their own fields to represent its information so after this we are going to use this nested gmf json data as a demo data source after we attach this json file to the data source i attached it here we must mark the data in single column here as true only when this field is true then epo app would process this payload as a nested payload otherwise it will process it as a flat payload after this data source is loaded each raw json data will be stored in the u underscore column which is the first column here each row represents a computer json object in the raw json array we just saw as we can see here we have three computers here and there is exactly the json structure we have so after data source is set up customer can go to etl app and start creating a new map based on this data source we just saw for the first step to fill out basic details there is no difference comparing to the original interaction flow this is the first step customer can simply fill out application information and the choose corresponding data source as before and click save here next is preview and the prepared data page one very important thing for nested payload is to efficiently navigate the nested data in prepared data page we provide customer two ways to navigate in nested data one is a tree view here the other one is a collection view both views sit in the left side panel customer can open and close this left panel through this show data structure check box you uncheck it it'll hide it check it it'll go back again first let's talk about the tree view one typical way for customers to navigate their data is to view a specific object in a specific level for example view the software installed of the second computer thus inside treeview will represent the exact nested data as a peel tree the light green peels represent the level objects such as a storage disk here or a software installed each level has ypls to represent the child records inside this level as here the white pills customer can expand the child pairs child pills by clicking it and then you can see the child records level again if i click it you see a live green peels as the level objects again when customer clicks appeals we also provide that breadcrumb to notify the object path they are currently at so currently i'm at the first level object if i click the software you see i'm going to the first computer software level as uh here in the breadcrumb clicking the breadcrumb itself can also navigate to corresponding levels which function same as clicking pills so if i click the object it's going to go back to the computer level again each time customer clicks a level pills or breadcrumb links the right side transform table which is this table will refresh to show all object information inside that level we also added listed object columns inside the transform table to make customers know how many nested objects they have for this records as you can see here we have the subtitle as a nested object you can see it's nested the software which is here and the storage device so on and then we have links here clicking this number link can also redirect you to corresponding level and the peel tree and the breadcrumb will update accordingly if i click the software here it will go to the first row which is first computer software level and then we can see all the software information we have here customer can also group by or search their data they function same as standard platform table guru by and the search features for example if i want to group by the computer mac address here i just click this hamburger and we see the screw by computer mac address and clicking it it will group by this and the search function has a standard platform we have this magnifier button let's search for the third row as f4 and let me click enter we see that we search by the f4 this row shows up next is a collection view in this field we provide customer holistic view of objects they have in one level for example customer might want to see all the software they have no matter those software from first computer or second computer the left side now represents all the levels inside nested data when clicking any of the level pills the right side table will show all objects inside that level as here we are showing the software of the all the software we have inside this data also in order to allow customer to know which parent object these records belong to for example which computer this first row software belongs to we added a context columns in transform table to represent the parent objects as we can see here the first column we have the subtitle as context column which shows the parent information of all those rows information and by default we group the level objects by the top most parent they have which is computer object context column in this case when customers create transform operations in nested payload they can achieve exactly what they have been doing in black payload version and i have copied all the transform operations exactly from the flat payload version here as you can see and even better we added a few dot work pickers for customers to select fields we want to transform from different nested levels inside this speaker we will first show the topmost levels fields here this is a dot walker picker downward figure if i add it here we see that all these fields are from the first level and then we will we customer can click the downward row which is here you can see the right arrow here if i click this they click this they can navigate to other levels fields which is here it's from fields from software for example let's see take an example as a sam software source native key here in the flat pillow version customer has this same software as some key which is a concatenation between computers id and the software source native key and they are from different levels because they are computers and software levels previously they have to flatten the software data with computer data so they can create the same level level transform but always nested payload feature customer can just open the software level which is this level we currently have here choose a concatenation operation here select the first input column as a computer's udid which is id of computer as you can see here which is from the first level and then for the second column go to the dialog picker select software level and the software snk here then they can achieve a different level transform operation and just click apply it will give you results as a concatenation of the software of the computer's id and the software's source native key after finish creating transform operation we will go to the class map page i have already created the exact same mapping as a flat gmf mapping but with nested payload feature we can save a lot of extra mappings that exist in flat gmf let me go to the flat gm f1 this is right now i'm currently at the flat version of the jmf computer mapping so as you can see here the flat flat gmf data does strict operations to flat data so they need to add an extra computer class for each nested level class so they can be associated together for example for this disk class here they have to add this this computer and for the network adapter they have to add the network adapter computer just so that they can be associated together and also because the data is flattened they have to add the extra fields which is underscore type here to make every class a conditional class and they use this you underscore types value as a condition to give information about which data this this data should map to which class like here if your underscore type is disk they map to this class but with the nested feature here and i'm back to the new version of the nested payload versions mapping customers only need to map one single basic class of the computer the top level computer class and associate all nested class with this single computer as you can see here we have this software we don't have all those extra computer class everything is associated to the single computer another ui change is that when customer created a conditional class previously we only showed a few name in the condition builder which is here but now is nested payload customer needs to know which level of view belongs to thus we prefix the level name before the field name as you can see here we are prefix the object which is first level of computer to the before the field name these prefix will show up in the encoder query in conditional class console as well which is inside here other class mapping flow is exactly the same as before next will be the attribute mapping same as class mapping i have already created the exact same required ire attribute mapping as flat gmf mappings and but for demo purpose i did not add those non-required attributes first we introduce a new peel tree at right side to better represent all fields and their corresponding levels we also give each field a sample value to give customers a better understanding of what this field means at this right side as you can see here this computer ip address which is has this and this is a typical mac address customer can still drag these peers to the same db field they want to map in next payload feature we will highlight all the same same db field that can be mapped to when dragging peels so let me drag this one so you can see here the column here of the source recency times that step is surrounded with a green color and once customer maps their first attribute which is a computer's first level appears here inside this section they can only choose attributes from the level that the first map attribute belongs to if customer tries to drag appeal from other levels then we will highlight the same db field as error and show error notification so here we we already map all those fields to the first level computers fields so if i choose from different level for example software and if i try to map here you can see that it gives surrounded by red color and we give the error information say that this data is not allowed if customer wants to switch to use attributes from different level they have to delete all currently mapped fields and downstream reference and then we will allow them to choose any attribute from any level again additionally we provide customer an alternative way to select fields instead of dragging peers they can also open the new dot work picker to pick the fields from any level which is here this button and but the same rules apply to the dot work peak as well if customer already mapped you an attribute from a level which in this case will already map to the computer level then the attribute from other levels will be disabled so we see that those computer fields are enabled but other levels such as software printers are all disabled and you cannot select them final mapping step is a relationship mapping same as class mapping i have created the exact same relationship mapping as flat gmf but with nested filler payload feature because we no longer need the separate computer classes we mentioned that in the computer as a class mapping step for each of those nested class we can just simply create a relationship between the single computer class with all nested class let me compare this with the original jmf computer so right now i'm on the old flat jmf computer smacking and let's go to relationship page so you can see here because they have this extra you underscore type field so they have to make it a conditional relationship and you can see they map all those duplicate relationship of those this computer network level computer id address computer just so that the relationship could be associated together but here we only need to create three simple basic relationship with a single computer class and all those nested classes simple and clear another new ui change is that we prefix the level name before the field name in the conditional builder for conditional relationship mapping as well these prefix will show up in the encoder in conditional class cards as well so same as class mapping we map the level name before the field names and they will be show up in the conditional mapping class cards also because the data is nested now it's natural that we cannot create a relationship between two sibling levels for example here you can see the software and the storage disk siblings so if the storage uh siblings in this nested data tree so we cannot create relationship between them if i go to a relationship page and try to create a new relationship if i select the disk which is story disk and parent the software will not show up in the child and vice versa if i select software then the disk will should not show up in the child option after customers finish all the above mapping flow they can proceed to the preview and the rollback data page to check in certain nested data as before for demo in purpose i extracted three gmf nested computer so we can clearly see all the records are injected and associated correctly let's go back to the json data we have three computers two printers three software each computer has one network adapter one serial number one ip address and one disk so in total we have 20 records and the night relationship which is here you can see the number is correct and inside each class tab here we can see the reference fields are all correctly associated so here is the three computers we have inside this we can see that the disk is associated to those three computers and for ip address it's associated to the nic which is network adapter here and for network adapter inside the configuration item it's associated to three computers and we have those two printers here same serial number is associated to the computers the software is correctly injected into the sam software installation class and it's installed on those three computers and we have the night relationship created between the network adapter ip address and computer that's an end to end flow for nested payload feature you