How to link an ECC Queue record back to a specific Feature or Job - Support and Troubleshooting

How to link an ECC Queue record back to a specific Feature or Job Issue All jobs that run via a MID Server, and some that don't, pass messages via the ECC Queue table [ecc_queue]. The Agent Correlator field [agent_correlator] is used by code to keep track of the originating job.

Although this list is in no way comprehensive, it should help you link an ECC Queue record back to the job it came from, to understand the code responsible. The code involved in launching the probe, the Probe code that runs in the MID Server , and Sensor code back in the instance, is usually owned by the particular feature, and the MID Server platform is simply running it in a java thread.

Table of Contents IT Operations Management ITOM Visibility - Discovery / Service Mapping ITOM Visibility - Test Credential ITOM Visibility - Help The Helpdesk (HTHD) - Direct to Instance ITOM Visibility - Agent Client Collector - Visibility (ACC-V) ITOM Optimization - Cloud Discovery - Direct from Instance ITOM Optimization - Cloud Management ITOM Health - Event Management ITOM Health - Agent Client Collector framework (ACC-F) / Visibility (ACC-V) / Monitoring (ACC-M) ITOM Health - Health Log Analytics ITOM Health - Operational Intelligence Integrations Orchestration Outbound REST / SOAP LDAP Import Set / JDBC Custom JavascriptProbe IntegrationHub Syslog Integration AttachmentCreator SOAP web service Export Sets 3rd party: IT Service Management Security Incident Response Microsoft Exchange Online for Security Operations Agent Client Collector for Security Incident Response Employee Document Management Now Platform Source Control MID Server IT Operations Management ITOM Visibility - Discovery / Service Mapping ECC Queue Topics:

ADMEPowershell a Windows PowerShell based probe specifically for Application Dependency Mapping. APIProxyProbe CimProbe Command Runs the command in the 'Name' field from the shell (Linux) or Command Prompt (Windows) of the MID Server host. CommandPipeline DNS DNSLookupProbe DNSNameResolver HorizontalDiscoveryProbe Runs the Discovery or Service Mapping Pattern specified the name field, on the MID Server, for the target server in the source field HTTPClassyProbe JavascriptProbe Runs the Javascript from the 'script' parameter of the payload. Usually calls a MID Server Script Include [ecc_agent_script_include]' MultiProbe A probe containing multiple other Discovery probes PatternDebuggerProbe Powershell Run a Powershell script on the MID Server, which may in turn do things to a target RCASmcPurgeProbe ServiceDiscoveryProbe ServiceWatchTraceRouteProbe Runs ping then traceroute from target host to IP as part of network path determination process. Shazzam The Port scanner for Discovery SNMP SSHCommand TrafficBasedDataCollectionUnix URLCertificateScanProbe Certificate Discovery. Name is "URL Certificate Scan" VMWareProbe WindowsCommand Runs a command on the command line, of the target server in the source field. WMIRunner Retrieves data from Windows Management Instrumentation (WMI) namespaces given in the parameters, of the target server in the source field. Agent Correlator:

Sys ID of the Discovery Status record [discovery Status] Source:

Varies according to Probe. An IP Address can be linked to a specific Discovery Device History [discovery_device_history] record within the Discovery Status to find the CMDB CI. Sensors:

Discovery - Sensors ITOM Visibility - Test Credential ECC Queue Topics:

CommandPipeline Source:

IP Address of the host that the credential is being tested against Payload

Will contain "type="TestCredentialCommand"" ITOM Visibility - Help The Helpdesk (HTHD) - Direct to Instance ECC Queue Topics:

WMILoader (input only) Agent:

wmi Name:

wmi.script Source:

Hostname of the Windows computer that sent the record to the ECC Queue Sensor Business Rule:

ECC Queue Reader - Async business rule which uses GlideappWMILoader in WMILoader.java. ITOM Visibility - Agent Client Collector - Visibility (ACC-V) See ACC-F in general

ITOM Optimization - Cloud Discovery - Direct from Instance ECC Queue Topics:

RESTProbe SigningRESTProbe AWSRESTProbe Agent:

mid.server.NODE_AGENT Agent Correlator:

TBC ITOM Optimization - Cloud Management ECC Queue Topics:

TBC Agent Correlator:

TBC ITOM Health - Event Management ECC Queue Topics:

ConnectorProbe MIDExtension:VCenterExtension MIDExtension:TrapListenerExtension MIDExtension:PipelineManager MIDExtension:WebServerExtension Agent Correlator:

Empty Source:

For Connectors, the name of the Connector Instance [em_connector_instance] Name:

For MID Server Extensions, name of the Extension, and Context Name and Sys_id [ecc_agent_ext_context] ITOM Health - Agent Client Collector framework (ACC-F) / Visibility (ACC-V) / Monitoring (ACC-M) ECC Queue Topics:

MonitoringProbe Name/Source=on_demand_request Payload will contain the name of the Check to run on the ACC, and the Agent ID(s) to run the check on. Payload is the only field with information on which feature, check, Agent is involved. Inputs are often too big for the payload field so end up in a payload.txt attachment, meaning payload contains filter on ecc queue lists will miss out records. The attachment needs opening to see what the input was for. Name/Source =config_publish/policy_deleted Updates running_checks_num in sn_agent_ci_extended_info record. Sensor business rule: MonitoringProbeSensor Name/Source = auto_mid_selection "Try redistributing connected agents" UI Action on MID Server form. Sensor Instance: AgentNowResponseProcessor business rule, using AgentNowHandler script include, processEccRecord function. sn_agent_check_type records map the Check Type specified in the payload to the Script Include that is run as the sensor for that check type. e.g. "EnhancedDiscovery" type uses EnhancedDiscoveryHandler script include. "ACC Spoke" just updates a record in sn_acc_spoke_action_response. DataInputMarkerProbe / DataInputExamplesProbe /DataInputConnectorPortCheckProbe Used by Log Analytics MIDExtension:MonitoringExtension Stop and start extensions Sensor:

All MonitoringProbe inputs, for any feature, are initially handled by the AgentNowResponseProcessor business rule. sn_agent_check_type records map the Check Type specified in the payload to the Script Include that is run as the sensor for that check type. ITOM Health - Health Log Analytics ECC Queue Topics:

queue.log_streaming.stats collects streaming sources stats data on the MID and reports to the instance. This data will be available in the instance sn_occ_log_streaming_sources table. queue.log_analytics.stats collects metrics from the log analytics services on the MID and reports to the instance. Those metrics will be reflected in the instance xmlstats.do MIDExtension:DataInputWrapperExtension Stop/Start/etc. of the MID Server extension DataInputTestConnectionProbe Tests a Data Input. DataInputMarkerUpdate Periodic Inputs from the Marker Service For ACC-L, see the ACC section.

ITOM Health - Operational Intelligence ECC Queue Topics:

MetricConfigProbe - Synchronizes Metric Configuration Rules to MID Server ( Docs link) TBC

Integrations Orchestration ECC Queue Topics:

JDBCOrchestrationProbe JDBCOrchestrationProbeCompleted Powershell SSHCommand SystemCommand CommandPipeline Used with SFTP Activity Command runs a local command on the MID server. DNSNameResolver resolves a fully qualified domain name (FQDN) into an IP address SCPCommand copies files securely from one machine to another SSHCommandLong executes long running shell scripts from a command line after logging in to a target machine via SSH Agent Correlator:

The Workflow Context [wf_context] record Sys ID, prefixed with "rba.". Source:

Powershell: The target hostname JDBCOrchestrationProbe/JDBCOrchestrationProbeCompleted: Empty Sensor Business Rule:

Automation - Sensors Quebec documentation: Deprecated: Orchestration activities: Probes used by Orchestration

Outbound REST / SOAP ECC Queue Topics:

RESTProbe (will run as a MIDWorker thread in the MID Server, as RESTProbe is not an actual Probe) SOAPProbe Agent Correlator:

By default empty, unless set using the setEccCorrelator() method of the API. Source:

The REST Endpoint URL, unless overridden by the setEccParameter() method of the API. Agent:

mid.server. - This will be the MID Server that processes the job. The following 3 values are for jobs run directly by the instance, directly to the endpoint, without going via a MID Server:- RESTClient - This is processed in the instance by the "RESTClient" business rule, triggered by the insert of the output ECC Queue record. SOAPClient - This is processed synchronously on ecc_queue insert, by the instance, by the "SOAPClient" business rule, which calls SOAPMessageV2() and .execute();. If system property glide.processor.debug.SOAPProcessor=true, then app node localhost log will give debug of the outbound request being executed by that business rule. Search for "OUTBOUND_HTTP:" and "SOAP Msg Outbound" in the logs. You can also see the logging in the instance table System Logs --> Outbound HTTP Requests. mid.server.NODE_AGENT" also indicates a request from the instance, not via a MID Server. Sensor business rule:

Generally custom, or specific to the endpoint. "Process Get Stock Quote" is an out-of-box example sensor for a SOAPClient direct-from-instance job. KB0995097 Troubleshooting localhost logs for Outbound Web Service issues (internal)

LDAP ECC Queue Topics:

LDAPConnectionTesterProbe LDAPListenProbe LDAPListenProbeCompleted LDAPProbe (expect 1 output, but 1 or more inputs) LDAPProbeCompleted LDAPProbeError LDAPBrowseProbe Agent Correlator:

A random GUID, but can be used to group records from a LDAPProbe run together Agent:

This is usually the MID Server name, but LDAPProbe and LDAPListenProbe topic inputs will have "LDAPListenProbeResult" and "LDAPListenProbeResult" for the inputs containing the data. Name:

For LDAPProbe and LDAPListenProbe inputs, this is the upper Row number for the batch of rows included in the input, or the last row if it is the last input. Source:

Sys ID of the LDAP Server record [ldap_server_config] Import Set / JDBC ECC Queue Topics:

JDBCProbe - expect 1 output, but 1 or more inputs JDBCProbeCompleted JDBCProbeError - Used to tell the MID Server to stop running a JDBCProbe when the Data Source times out on the instance side Agent Correlator:

A random GUID, but can be used to group records from a Import Set run together Agent:

This is usually the MID Server name, but JDBCProbe topic inputs will have "JDBCProbeResult" for the inputs containing the data. Name:

For JDBCProbe inputs, this is the upper Row number for the batch of rows included in the input, or the last row if it is the last input. Source:

Sys ID of the Data Source record [sys_data_source] Sensor Business Rule:

JDBCProbeSensor - responds to JDBCProbeCompleted topic. Runs async as "ASYNC: JDBCProbeSensor" in scheduler workers, using process() in Script Include JDBCProbeSensor. Custom JavascriptProbe ECC Queue Topics:

JavascriptProbe Source:

Empty by default, unless overridden by setSource() API method. The payload contains a "script" parameter, which may identify the MID Server Script Include used by the probe.

IntegrationHub Integration Hub Flow Designer - Retry Policy

The ecc-queue records don't contain any data/results. When the IPaaSActionProbe runs, it will use instance Flow Engine REST APIs for instructions on what to do, and for updating the flow context with the results/data. To debug issues with this probe's execution may need instance-side appnode localhost logs for the API_INT semaphore transactions made during the execution.

ECC Queue Topics:

IPaaSActionProbe Agent Correlator:

Flow Context ID [sys_flow_context] Name:

The MID Server's name, when first inserted. This may not match the MID Server that actually ran it if there was any MID Cluster failover to another MID Server. (MIDSender.java legacySend) Source:

Flow Context ID [sys_flow_context] Sensor business rule:

IntegrationHub - Sensors IntegrationHubResponder Reschedule Flow Instance APIs:

/api/now/hub/plan/[sys_flow_context] Note: Steps running PowerShell scripts will use the same low-level MID Server platform code for executing the script as Discovery and Orchestration. Enabling logging for the powershell step , and mid.log.level=debug mid server parameter will show exactly what PSScript.ps1 is doing.

Syslog Integration Syslog probe

ECC Queue Topics:

Syslog Payload:

Contains parameters for the syslog host and message Note: As of the Utah version, there is minimal logging and no additional debug logging that can be enabled for this probe. To investigate what this probe is actually doing may require a network packet sniffer, such as WireShark installed on the MID Server host.

AttachmentCreator SOAP web service AttachmentCreator SOAP web service

ECC Queue Topics:

AttachmentCreator Name:

The name of the file being attached, and its content-type. Source:

" : " of the record the attachment is to go on. Payload:

A base64 encoded string, representing the binary object to be attached. Sensor business rule:

AttachmentCreatorSensor Export Sets ECC Queue Topics:

StreamPipeline (output) ExportSetResult (input) Payload parameters:

stream_relay_response_topic=ExportSetResult stream_relay_source attachment_sys_id = sys_id of the sys_attachment record being exported stream_relay_transform attachment.table_sys_id = sys_id of the ecc_agent_attachment record. The source field of that record references the Export Set [sys_export_set] record. stream_relay_sink path = file name/path Community post: Export a record attachment to a mid server's export folder

3rd party: JDBC File Loader ECC Queue Topics:

JavascriptProbe Name:

JDBCFileLoaderProbe This is the MID Server Script Include name. Source:

sys_id of the file source record Sensor business rule:

JDBCFileLoaderSensor Async business rule, using script include ImportSetUtilPlus File Builder FTP This was a ServiceNow Professional Services product, developed by Jason Petty in 2014, shared on ShareNow as an Update Set. As of 2024, a Video Tutorial remains on youtube explaining the definition tables and forms used to set this up. The code triggering this probe will usually be a call in a scheduled script to the FileBuilderFileCopyES script include.ECC Queue Topics:

JavascriptProbe Name:

FileBuilderFileCopy Remote File Importer This is originally written by partner Cloud Sherpas (now Accenture), and shared on the developer site.

Developer site: Remote File Importer (Import Files from a MID Server) Community site: Getting Error in "Input ECC Queue" as "No sensors defined". I am getting this when trying to execute one of the "Remote file import". KB1641779 Unable to import files from Mid Server after upgrade to Washington DC ECC Queue Topics:

JavascriptProbe Name:

CSRemoteFileImport Payload:

targetImportSet - An instance table name filePath - File path on MID Server disk script = var remoteFileImport = new CSMIDServerRemoteFileImport(); remoteFileImport.getRemoteFileAndUploadToInstance(); The probe code is in MID Server Script Include "CSMIDServerRemoteFileImport". The instance Script Include "CSRemoteFileImport", UI Page "run_remote_import" and UI Action "Import Now" are also involved.

IT Service Management Security Incident Response The Security Incident Response application includes third-party integrations.

ECC Queue Topics:

RESTProbe Agent Correlator/Source/Payload:

TBC Microsoft Exchange Online for Security Operations Docs: Microsoft Exchange Online integration (Rome)

The integrations are implemented by Powershell Orchestration Workflow activities:

Diagnose Compliance Search Create Search on Exchange Online Check Status of Exchange Online Search Search Threat Email in O365 MID Selection Criteria:

Capabilities: PowerShell, WinRM Basic Authentication Applications: Orchestration ECC Queue Topics:

Powershell Name:

Windows - PowerShell Payload:

MIDScriptFile parameter will give the powershell script name, that will link it back to the specific activity. Agent Correlator:

The Workflow Context [wf_context] record Sys ID, prefixed with "rba.". Source:

The target hostname Sensor Business Rule:

Automation - Sensors Agent Client Collector for Security Incident Response This app uses Flow Designer workflows, and the Agent Client Connector Spoke app for IntegrationHub.

Employee Document Management ECC Queue Topics:

StreamPipeline (output) - created by ef_LocalFileCapture script include, importFileUsingMid function. EmployeeFileImport (input) Payload parameters:

stream_relay_response_topic=EmployeeFileImport stream_relay_source connection.hots, connection.port, crdential_tag, path, type stream_relay_transform attachment.table_sys_id, attachment.table_name, attachment.file_name, attachment.content_type, type=AttachmentSink stream_relay_sink path = file name/path Sensor business rule:

Process local file import uses processLocalImportResult function in ef_LocalFileCapture script include. Adds result to sn_hr_ef_import_staging record. Attachment data is not in the payload, which is inserted directly into the instance attachment table by the probe, using /now/attachment/file REST Attachment API . Now Platform Source Control ECC Queue Topics:

ImportApplicationProbe SourceControlProbe RemoteCloneProbe ExportApplicationProbe ImportNewApplicationProbe CreateTagProbe ExportNewApplicationProbe TestConnectionProbe RefreshRepositoryProbe CreateBranchProbe Name

Source Control Sensor Business Rule

Source Control Response Async business rule using MIDResponse.java and GitECCProcessor.java, where invokeOperation calls a function specific to the operation. . MID Server ECC Queue ECC Queue Topics:

queue.processing inputs are used to notify the instance which outputs are 'processing', and subsequently 'cancelled' or 'processed'. Sensor business rule: "ECC Queue - mark outputs state" - Updates the state of the output records mentioned in the payload. queue.stats Each MID Server reports every 10 minutes on currently running threads, and performance metrics. Sensor business rule: "MID - Process XMLStats" - Updates ecc_agent_thread records, JVM version in ecc_agent, and via AgentMetrics script include updates ecc_agent_counter_metric, ecc_agent_scalar_metric, ecc_agent_rgr_metrice, cc_agent_memory_metric. Kubernetes Deployment Agent:

This needs to be a MID Server that is set up as a Kubernetes Deployment MID Server, with the "MID Server Management" application.

ECC Queue Topic:

KubernetesOperationProbe Name:

Name field of the request's mid_k8s_deployment record. Payload:

deployment_request_id is the sys_id of the mid_k8s_deployment record. deployment_request is BASE64 encoded, and then encrypted as well. Sensor Business Rule:

KubernetesOperationResponder Updates the deployment request mid_server_deployment record state. System Commands ECC Queue Topics:

SystemCommand Source:

The name of the MID Server internal system command. These include: restart Restarts the MID Server. restartService Restarts the MID Server Service. This will completely stop and then start the Windows service. stop Shuts down the MID Server service threaddump Writes a Java Thread Dump to the wrapper.log gc grabLog Retrieves the file in the name field. getResourceState deleteLog pause / unpause Pauses the MID Server so it will not run jobs, however most system commands will continue to be run. upgradeNow Prompts the MID Server to check if it needs to upgrade now, rather than wait for the hourly scheduled job to check. clear_queue load_properties load_ip_access IP Address Access Control Plugin. This will apply any rules that have "Enforce on MID server" checked to the MID Server. cancel_discovery When a Discovery Status reaches max runtime, or is manually cancelled, to notify the MID Server to cancel any remaining probes. Broadcast to all MID Servers, regardless of whether that are running Discovery. agent_correlator= updateConfig For pushing changes to parameters/properties to a MID Server. Parameters will also be written to config.xml. status invalidate_cache script Used by MID Server Scripts - Background page, to run any javascript in the mid server, and return the output credentials_reload Triggered by a change to any discovery_credentials record. Updates the MID Server' local cache of its credentials. privileged_command_reload automation_encryption_keys_reload probe_cache range_cache Clears the Discovery IP Range Cache. Triggered by changes to discovery_range table. delete_mid_keypair Re-Key and Invalidate acls_change mibs_reload Re-Synchronises all SNMP MIB files from ecc_agent_mib FileChange Re-Synchronises Files to the disks of all MID Servers. The Name value is the table where a synchronized file change has occurred, e.g. JAR Files, Script Files/Includes. resetQueryWindow CitChanged CustomOperationChanged CustomParsingStrategyChanged LibsApplCategoryChanged LibsDeviceInfoCategoryChanged PatternExtensionChanged MetadataRulesChanged ExtCommandChanged pauseMid Stops the MID Server from running any jobs except system commands. resumeMid Allows the MID Server to run any jobs. autoUpgrade Prompts the MID Server to check if it needs to upgrade now, rather than wait for the hourly scheduled job to check. grabNdl trackedFileDefinitionChanged clear_cookies Clears the cookie cache, which is used by the SOAP client for making requests to a ServiceNow instance, allowing the MID Server to connect to to a different app node. installNmap / uninstallNmap Installs/uninstalls NMAP on the MID Server host, which is used by Credential-less Discovery. service_account_reload file_discovery_whitelist file_discovery_refresh connection_cache_update environment_clear mid_status_change