DUPLICATE_PAYLOAD_RECORDS Found duplicate items in the payload (index xxand xx) - Support and Troubleshooting

DUPLICATE_PAYLOAD_RECORDS Found duplicate items in the payload (index xxand xx) Issue Executing Discovery on the Network Router devices fails with the below error (Multiple related errors)

Pattern logs

2023-07-18 00:41:37: The following item indices were removed from payload: [17, 18] 2023-07-18 00:41:37: : There is problem to build the Sanitized Payload: Identification Engine errors : See Identification Logs for more info.

Update failed with error: Commit was not attempted due to other errors .,Update failed with error: Commit was not attempted due to other errors., Insertion failed with error: Commit was not attempted due to other errors .,Insertion failed with error: Commit was not attempted due to other errors., Update failed with error: Commit was not attempted due to other errors .,Insertion failed with error: Commit was not attempted due to other errors., Insertion failed with error: Commit was not attempted due to other errors . System logs

{"items":[{"className":"cmdb_ci_ip_router","sysId":"Unknown","identifierEntrySysId":"Unknown","errors": [{" error":"ABANDONED","message":"Abandoned due to too many errors "}],"identificationAttempts":[],"info":[],"errorCount":1,"warningCount":0,"inputIndices":[0],"mergedPayloadIds":[],"markers":[]},{"className":"cmdb_ci_ip_address","sysId":"Unknown","identifierEntrySysId":"Unknown","errors": [{" error":"ABANDONED","message":"Abandoned due to too many errors "}],"identificationAttempts":[],"info":[],"errorCount":1,"warningCount":0,"inputIndices":[1],"mergedPayloadIds":[],"markers":[]},{"className":"cmdb_ci_ip_address","sysId":"Unknown","identifierEntrySysId":"Unknown","errors": [{" error":" ABANDONED","message":"Abandoned due to too many errors "}],"identificationAttempts":[],"info":[],"errorCount":1,"warningCount":0,"inputIndices":[2],"mergedPayloadIds":[],"markers":[]}, com.snc.sw.topologymanager.drp.IdentificationEngineDuplicationHandler.matchSingleCi(IdentificationEngineDuplicationHandler.java:206) com.snc.sw.topologymanager.drp.IdentificationEngineDuplicationHandler.execute(IdentificationEngineDuplicationHandler.java:104) com.snc.sw.resulthandlers.AbstractDiscoveryResultHandler.runIdentificationEngine(AbstractDiscoveryResultHandler.java:150) com.snc.sw.resulthandlers.HorizontalDiscoveryResultHandler.runIdentificationEngine(HorizontalDiscoveryResultHandler.java:1716) com.snc.sw.resulthandlers.HorizontalDiscoveryResultHandler.jsFunction_runIdentificationEngine(HorizontalDiscoveryResultHandler.java:1590) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.base/java.lang.reflect.Method.invoke(Method.java:566) org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138) org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:677) org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2649) org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) org.mozilla.javascript.gen.discovery_sensor_2f32f7899f230200fe2ab0aec32e706a_33._c_anonymous_2(discovery_sensor.2f32f7899f230200fe2ab0aec32e706a:185) org.mozilla.javascript.gen.discovery_sensor_2f32f7899f230200fe2ab0aec32e706a_33.call(discovery_sensor.2f32f7899f230200fe2ab0aec32e706a) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2734) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2657) org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) Errors from the IRE Simulator

2023-07-25 23:27:00 : Info : identification_engine : logId:[6dc400c1879c] DUPLICATE_PAYLOAD_RECORDS Found duplicate items in the payload (index 15 and 16), using className [dscy_route_interface] and fields [dest_ip_network,mac_address]. Remove duplicate items from the payload 2 : 2023-07-25 23:27:00 : Info : identification_engine : logId:[6dc400c1879c] DUPLICATE_PAYLOAD_RECORDS Found duplicate items in the payload (index 16 and 15), using className [dscy_route_interface] and fields [dest_ip_network,mac_address]. Remove duplicate items from the payload 3 : 2023-07-25 23:27:00 : Error : identification_engine : logId:[6dc400c1879c] Detected error while processing payload from ServiceNow. Using options: Release Observed on

Instance: Tokyo Version Discovery and Service Mapping Patterns plugin: 1.6.0 Cause For the payload where it was successfully discovered :There was this item which was duplicated once in the payload (so items with index 14 and 15 were the same)

{ "className" : "dscy_route_interface", "values" : { "dest_ip_network" : "0.0.0.0/0", "install_status" : "1", "name" : "0.0.0.0/0[Direct]", "type" : "3", "sys_class_name" : "dscy_route_interface" }, "internal_id" : "0dad18c0-483e-4194-b2e3-d694f3ccbad0" } While discovering the device, in the "Identification Engine Payload Sanitization" step of the Payload processing the 15th index was removed from the payload, there are no more duplicate Items and the CI is successfully discovered and updated.

While for the payload where we have errors : the same item (index 14) was duplicated 4 times in the payload (Item 14, 15, 16 and 17 are the same)

{ "className" : "dscy_route_interface", "values" : { "dest_ip_network" : "0.0.0.0/0", "install_status" : "1", "name" : "0.0.0.0/0[Direct]", "type" : "3", "sys_class_name" : "dscy_route_interface" }, "internal_id" : "0dad18c0-483e-4194-b2e3-d694f3ccbad0" } In the "Identification Engine Payload Sanitization" step of the Payload processing only the 15th and the 17th index was removed from the payload but not the 16th, raising then an error here.

Resolution The solution was to remove duplicate item from the payload, to do this, add a step to the Network Pattern just after the router exit interface insertion step, where we remove the duplicated item from the payload according to the KB bellow :

IRE Identification DUPLICATE_PAYLOAD_RECORDS error