How to write an RTE Consumer for Stream ConnectSummarySteps on how to write an RTE Consumer for Stream ConnectReleaseAll releases.Instructions Listing the main components required while setting up an RTE Consumer : 1. SOURCE MESSAGE: The message payload produced to the topic. 2. TARGET TABLE: The ServiceNow table to import the records into via RTE. Example table: Incident Table 3. The Mapping strategy/requirement of the fields: The importing of messages requires a mapping from the source message to the columns in the import table For ex. I want to map the following in incident table to the complex string formed from the source message Short description (short_description) <---------> “Status description :” + statusDescription Description (description) <---------> taskId Example JSON message with nested fields: { "context": { "taskId": "281902712", "taskType": "Error task", "businessContext": { "accountNbr": "6368N4164" } }, "statusDescription": "Pended", "updatedAt": "2024-11-05T15:17:28.679+0000" } ## NOTE Even if all mandatory fields are not mapped, the import should work. RTE Records The following is the list of records that need to be created on the platform for a RTE Consumer. ETL Definitions (sys_rte_eb_etl_definition) Add the following details NameDescriptionCopy on empty fieldsExecute on before scriptExecute on after script ETL Definition record above: Once you save the records you will have two related lists link, associated for ETL Entities and RTE ETL Mappings which are discussed below ETL Entities: (sys_rte_eb_etl_entity) ETL Entities are a representation of any entity you can create out of the data. It is intended to create source entities, temp entities and target entities. For each ETL entity multiple entity fields can be created which will hold the actual data value. Source Entity: Entities that you create from your source, here source is the JSON message, for RTE Consumer, you do not need to fill in the Table and Path. Once you save the page with the name of the entity, a related list for ETL Entity fields (sys_rte_eb_etl_field) will appear.Target Entity: Represent a Entity for a Servicenow table and the fields you want to map finally in the Servicnow table column. Temp Entity: These are intermediate entities which are used to do any operation like concatenating source fields or writing a custom script.a. In the temp entity it is better to add a list of entity fields to map from source fields and do operations on the source entity fields and create separate target entity fields which to map directly to the fields in the target entity, Entity fields: These are the actual fields in your JSON message which will be mapped from the source message to contain the data from the JSON message. To create an entity field from a nested key in JSON message, add the Field/Path as, the dot walk path from the JSON message for example in above source message to add an entity for accountNbr give the field/path as context.businessContext.accountNbr ## NOTE: Check for spaces in names, NO TRAILING SPACES especially in any field/path and Names for the entity fields Source ETL Entity record created above Example for Temp ETL Entity is given in the following screenshot below where sourcestatusDescription, sourceTaskId, sourceTaskType are intended to be mapped directly from the source entity fields above and the targetDescription and targetShortDescription will be mapped directly to the entity fields for target fields. RTE Entity Operations : All operations required to get the desired mapping from source data to target data should be created as RTE Operations in the for the temp entity. RTE Operations can be done on fields of the same entity only. Giving a few examples of the RTE Entity Operations below. Example 1: COPY (sys_rte_eb_copy_operation) - A simple operation is to copy a source field to a list of target fields Example 2: Script with Multiple Inputs ## NOTE Always refer the fields in script via their field/path, the entity field Name is sourcestatusDescription and Path is sourcestatusDescriptionSymbol so use the sourcestatusDescriptionSymbol for accessing the input in the script below. 3. Target Entity: Example of the Target ETL Entity here please select the desired Servicenow table to map the fields to and also give the path as the name of the table Mappings: (sys_rte_eb_entity_mapping) The mapping or essentially copying of field data from one entity to another. Added screenshots for mapping of data fields from source to temp and then temp to target entity. SOURCE TO TEMP MAPPING: Directly mapping the entity fields from source entity to temp entity 2. TEMP TO TARGET MAPPING: Directly mapping the entity fields from temp entity to target entity. ## NOTE You can import to multiple tables from the same RTE configuration by creating multiple target entities. ## NOTE If the input needs to have "." in a key, bracketed form should be used. ## NOTE: No import set / staging table needs to be created to use an RTE Consumer, it uses a virtual staging table for performing the transform. Robust Import Set Transformer Record. There are two ways to create it : 1. Robust Import Set Transformer Record via DataSources [Recommended] Steps: Create a json file with the json message surrounded by an array [<your json message>] Go to the sys_data_source table and add a new File Type Datasource with JSON Format and select the Data in single column option as mentioned in the screenshot below and attache the above file created. Give a desired import table name (In happy path scenario this table will not be populated for RTE Consumer) Add the Robust Transformer record from the related list, the source table gets populated automatically as the import set table, select the Transformer Definition you created above. 2. Robust Import Set Transformer Record vis script (sys_robust_import_set_transformer) ## NOTE Here source table record is mandatory so you can add the RTE Consumer via background script. The source table is not used for RTE Consumer in successful transformation but it is used for error scenarios. var gr = new GlideRecord("sys_robust_import_set_transformer"); gr.initialize(); gr.setValue("source_table","global_ihub_import_4941761743621210bbc51201018f276"); gr.setValue("name","json to incident"); gr.setValue("robust_transform_engine","sys_id_of_the_ETL_Definition"); gr.insert(); RTE Consumer Add a RTE consumer (sys_kafka_rte_consumer) by going to Nav menu and typing ETL Consumer. Fill in the details as mentioned below.