What is IH-ETL

• Create and manage ETL transform maps
• Integrate third-party data into the CMDB without compromising the integrity of the CMDB
• Simplified user interface that guides you through the integration process end-to-end
• Includes test integration run of sample data

Associated Components

• Robust Transform Engine (RTE): Used to transform raw source data that is stored in staging tables, into the data that is mapped and integrated into the CMDB. RTE uses ETL transform maps that were created for the integration during data transformation.
• Identification and Reconciliation engine (IRE): Used as a centralized framework for identification and reconciliation processes across different data sources. IRE processes help maintain data integrity in the CMDB.

IntegrationHub ETL uses RTE and IRE which work together to process and integrate data. Data is first imported from a data source, and is then stored in temporary staging tables in Import Sets systems. Using the data in the staging tables and the ETL transform map created by IntegrationHub ETL, RTE creates IRE payloads which are then processed by IRE. IRE applies reconciliation processes to avoid potential problems such as duplicate CIs, ensuring that the CMDB remains healthy, and then integrates the resulting data into the CMDB.

IH-ETL Workflow

At the high level, to create an ETL transform map you need to:

• Setup a Data Source configuration in standard platform UI to allow for data ingestion into the platform
• Input your transform map’s basic detail such as name, discovery source etc. in IH-ETL UI
• Preview your data and apply data transform operations (e.x. copy, to lower/upper case) if necessary in IH-ETL UI
• Define the CMDB class, field and relationship mapping plan in IH-ETL UI
• Run sample integration and preview how the data looks like after inserting into CMDB in IH-ETL UI
• After running sample integration, choose to retain or rollback the sample run data in IH-ETL UI

How to install IH-ETL

IH-ETL is automatically entitled on all sub-prod instances, all ServiceNow instances and all ServiceNow developer portal instances. If you need to entitle on your instance, please go to ServiceNow store and download the latest IH-ETL store app.

Roles Required

Users with the cmdb_inst_admin role can use IntegrationHub ETL to create integrations, or customize a pre-existing integration provided by ServiceNow or a vendor at the ServiceNow Store. A vendor can create a new integration and provide it as an application for anyone to use.

Terminology Definitions

Discovery source

• A choice list value on a Configuration Item that is used to identify the source of ingested data.

Data Source

• The definition of how data is ingested into the integration.

Staging Table

• A table defined to accept the incoming data. Columns can be dynamically assigned by loading data through the data source. Any column created in this way has the prefix “u_”.

Import Set

• The record that encapsulates a single ingestion of data. It relates to multiple rows in the staging table and references the Data Source.

Robust Transform Engine (RTE)

• The mapping and transformation engine that takes in the definition for how data should be converted from an object to an IRE payload.

Identification and Reconciliation Engine (IRE)

• The engine that inserts records into the CMDB in the correct classes. It also provides functionality to deduplicate and order inserts to avoid duplicate data or incorrect updates.

Additional Resources

Landing page

IH-ETL product documentation

Now Learning Course – K21 Import CMDB Data Safely Using IH-ETL

Integration Hub - ETL nested payload feature demo - NEW in Quebec

IH-ETL common error scenarios

Next Step

Prepare your data before creating transform map