MID Server FundamentalsAbout This document provides an overview of what the MID server is. It also covers the MID server most important features. There are links pointing to docs.servicenow.com for information on some of the topics, as they may change slightly depending on the version. On docs.servicenow.com select the version according to your instance. Additional Training MID Server Fundamentals MID Overview The Management, Instrumentation, and Discovery (MID) Server is a Java application that runs as a Windows service or UNIX daemon on a server in your local network. The ServiceNow®MID Server enables communication and the movement of data between a ServiceNow instance and external applications, data sources, and services. The MID Server initiates all communications with the ServiceNow® instance. This communication is recorded as records in the ECC Queue, which acts as the communication log between the instance and the MID Server. The MID Server picks up any work it has to do from the ECC Queue and returns the results of that work to the queue. Installation Install a MID Server on Windows Install a MID Server on Linux Configuration MID Server ParametersMID Server PropertiesSet the MID Server JVM memory sizeConfiguring MID Servers Upgrade MID Server UpgradesMID Server upgrade process - What actually happens when a MID Server upgrades itself? ECC Queue and AMB Channel The External Communication Channel (ECC) Queue is a connection point between an instance and the MID Server. Jobs that the MID Server needs to perform are saved in this queue until the MID Server is ready to handle them. The MID Server subscribes to messages published by the Asynchronous Message Bus (AMB), which notifies the MID Server that it has pending tasks in the ECC Queue. If a job exists in the ECC Queue for that MID Server, the MID Server sets the status to "I'm working on it." When finished working on a requested job, the MID Server reports back to the ECC queue with the results. The MID Server opens a persistent connection to the instance through the AMB)Client and listens on the /mid/server/<mid_sys_id> AMB channel. When an output record is inserted into the Queue [ecc_queue] table, an AMB message is sent to the MID Server's channel. The MID Server receives this message and immediately polls the ecc_queue table for work. The MID Server polls the ECC queue on the regular interval defined in the mid.poll.time configuration parameter, regardless of AMB message activity. The default polling interval is set to 40 seconds, but can be reconfigured. This polling of the ECC queue at a regular interval is done in case the AMB connection is dropped. ECC Queue Lifecycle One of the applications which make use of the MID server creates an ecc_queue output recordThe ECCQueueMonitor thread checks the ecc_queue for outputs in ready stateThe ECCQueueMonitor attempts to send the output message to one of the threadpools available in the MID server. The threadpool the output is sent to is based on the ecc_queue record priority field. There are three threadpools: InteractiveExpeditedStandard The ECCQueueMonitor thread sends an input back to the instance with a list of sys_id of the output records which were successfully allocated to one of the threadpool, this input will have topic=queue.processing. This input is processed and any outputs added to the MID server pools are set to "processing" on the instance. Note: if the MID server queue is full the output will not be picked up and thus remain as ready on the instance. The output message eventually leaves the threadpool and is assigned to a worker to be executed Note: The code to be executed will depend on the ecc_queue field "topic" The result is saved to one of the MID server ouput folders when complete, .\agent\work\monitors\ECCSenderThe ECCSender thread sends the results back to the instanceBusiness rule "ECC Queue - mark outputs processed" sets the output to processed based on the ecc_queue input field response_to. Also, one of the business rules in the ecc_queue table is triggered according to the condition and determines how the input is processed Note: The ECCQueueMonitor thread will run as configured in the poll time (mid.poll.time), or when triggered by the AMB channel. When an ECC message is inserted to the queue, an AMB message is published that notifies the MID server to adjust the polling frequency based on the priority of the message, and thus check on the ecc_queue for output records in ready state. For a more in depth review of the ECC Queue lifecycle see: How the ECC Queue works - from output ready, to input processedHow to link an ECC Queue record back to a specific Feature or Job Example ECC Queue Lifecycle HeartbeatProbe [INSTANCE] On the instance, the "MID Server Monitor" job creates an ecc_queue output record in ready state[MID] The ECCQueueMonitor check for outputs: ECCQueueMonitor.1 DEBUG: Monitor query: state=ready^queue=output^agent=mid.server.WIN-IHKAN2LQJE1^sys_created_on>=2021-08-10 10:26:38^ORDERBYpriority^ORDERBYsys_created_on [MID] we see the output is picked up: ECCQueueMonitor.1 DEBUG: Added processing (4): 8b5195cfdbf174104fa1c9db13961984ECCQueueMonitor.1 DEBUG: Event: MessageDispatchedEvent, message: HeartbeatProbe 8b5195cfdbf174104fa1c9db13961984 [MID] The "HeartbeatProbe" java class executes Worker-Interactive:HeartbeatProbe-8b5195cfdbf174104fa1c9db13961984 Worker starting: HeartbeatProbe [MID] The output of the "HeartbeatProbe" is sent to a file: Worker-Interactive:HeartbeatProbe-8b5195cfdbf174104fa1c9db13961984 Enqueuing: C:\ServiceNow\emprcoelhosilvaq\agent\work\monitors\ECCSender\output_0\ecc_queue.8b5195cfdbf174104fa1c9db13961984.xmlWorker-Interactive:HeartbeatProbe-8b5195cfdbf174104fa1c9db13961984 DEBUG: ** enqueued C:\ServiceNow\emprcoelhosilvaq\agent\work\monitors\ECCSender\output_0\ecc_queue.8b5195cfdbf174104fa1c9db13961984.xml [MID] The "HeartbeatProbe" processing is now complete: Worker-Interactive:HeartbeatProbe-8b5195cfdbf174104fa1c9db13961984 Worker completed: HeartbeatProbe time: 0:00:00.000 [MID] The output from the HeartbeatProbe is sent back to the instance: ECCSender.1 DEBUG: ** sending file C:\ServiceNow\emprcoelhosilvaq\agent\work\monitors\ECCSender\output_0\ecc_queue.8b5195cfdbf174104fa1c9db13961984.xmlECCSender.1 Sending ecc_queue.8b5195cfdbf174104fa1c9db13961984.xmlECCSender.1 DEBUG: Sent ecc_queue sys_id=475159cb1b7db0502fa799b61a4bcb71 [INSTANCE] We see the ecc_queue input with sys_id 475159cb1b7db0502fa799b61a4bcb71 created on the instance. Note: On the above example we see we can use the sys_id of the output, and of the input, ecc_queue records to track what happened in the MID server when this output was processed. ECC Queue Priority The MID server had three thread pools: Interactive Priority: 0Number of threads parameter: threads.interactive.max Expedited Priority: 1Number of threads parameter: threads.expedited.max Standard priority: 2Number of threads parameter: threads.max The MID server assigns work from the ecc_queue to the different thread pools based on the priority field of the ecc_queue record. The thread pools can be used to better manage work which will be sent to the MID server, and prioritize work that needs to be completed first. For example, scheduled discoveries use the standard pool by default. When running large discoveries there may be a backlog of work in the ecc_queue for the standard thread pool, while the expedited pool may have no backlog. If an integration making use of the MID server were to create an ecc_queue output for the standard pool it would wait in the same "line" as the discovery outputs. If such integration were time sensitive it would be best to send it to the Expedited pool. This way, such work would "skip the line" and not have to wait for the discovery to complete. ECC Queue File Size There are some properties which will control the maximum file sizes that the MID server will send to the instance, as well as the largest file sizes the instance will accept. Discovery and Orchestration mid.discovery.max_payload_size has a default value of 5000000 (bytes). Following error will be see in the input payload error: Payload length <Payload_Length> exceeded limit of <Max_Payload_Length> MID Server prooperties mid.eccq.max_payload_size is the maximum length of an ECCQ payload XML that a MID Server will send back to the instance. The default value is 20000000 (bytes). A result size in the MID server which is greater than this value will not be sent back to the instance. Instead, the payload part of the ecc_queue input will be modified to alert about the size issue. Example payload seen when the result is larger than what is configured in mid.eccq.max_payload_size: <payload><error>Payload size of <size_of_payload> bytes exceeded maximum of 20000000 bytes. Contents of the original payload were moved to <path> on the MID server. Maximum payload size can be set with mid config parameter: mid.eccq.max_payload_size</error></payload> System properties com.glide.attachment.max_size controls the maximum file attachment size in megabytes. An attempt to create an attachment bigger than what is configured in this property would trigger errors like: Attachment size exceeds the limit com.glide.attachment.max_get_size controls the maximum file attachment size in bytes and is used by discovery sensors. glide.soap.max_inbound_content_length property to control the maximum content length in MB which can be sent via SOAP to the instance. The following error will be seen when this value is exceeded: Request body exceeded max allowed content length Note: very large payloads could cause instance performance issues. Testing should be done in a non-production instance first. One or more of the above parameters may need to be adjusted in order to successfully send a result from the MID server to the instance. ECC_Queue and the API_INT The instance has a set of semaphores which will handle the requests from users, integrations, etc. Navigating to instance.service-now.com/stats.do will show you more information on the semaphores, sys_semaphore. When the MID server retrieves or sends ecc_queue records it uses the API_INT semaphore. For example (from the stats.do page): In the above image we see one of the API_INT semaphores in use by a MID server as it uploads an ecc_queue record. We also see the queue depth limit is 50. If requests to this semaphore do not perform well, this could lead to the queue depth limit being reached. If that ever happens, the requests to this semaphore would fail with 429 errors until there would be more availability in the queue again. Therefore, it is important that any integrations using the API_INT to perform well, otherwise it could block mid servers communication to the instance. MID Server error "Too Many Requests with code: 429" Credentials Authentication to the instance The MID Server uses basic authentication to connect to the instance using the credentials set in the config.xml file. After startup these credentials are encrypted in the file. Alternatively, starting in Quebec, the MID server can use mutual authentication to communicate to the instance. Enable MID Server mutual authentication Standard credential provider The MID server can run commands against devices. To successfully reach out to target devices and run commands it needs to have the proper credentials. Example applications which may need credentials: DiscoveryOrchestrationSpokesImports/JDBC By default, credentials are kept on the instance, encrypted, and synchronized to the MID server when necessary. For more information on credential encryption, order, and aliases see: Getting started with credentials For troubleshooting credentials please see: Credentials/Permissions troubleshooting on Discovery, Service Mapping, Orchestration External credential provider The credential values do not need to be kept in the instance. There is also the option to keep such credentials outside the servicenow instance. For such configuration, please see: External credential storageExternal credential storage configuration Status, Logs, and Troubleshooting Status The sys_trigger job "MID Server Monitor" runs every five minutes. This job creates an ecc_queue output records where topic = HeartbeatProbe for each MID server. The HeartbeatProbe output is processed by the MID Server and a HeartbeatProbe input is created. The heartbeat input updates the MID server last_refreshed field. The job also sets the MID servers status to "down" if there was no heartbeat reply since the last time a heartbeat was sent, and no recent activity is seen for such MID server. The detailed logic can be seen on script include MonitorMIDServer. See following kb for MID server status issues: MID Server Down troubleshooting Logs The MID server logs are kept in the .\agent\logs folder. The MID server writes information to two main logs: agent log, for example agent0.log.0 The agent log is where the MID server specific code writes messages to, info, debug, etc wrapper.log The wrapper log is where the java service sends logs to. This is also where the MID server upgrade service writes information to The MID server rotates the agent logs. The MID server will keep up to 10 agent logs, each with a size of up to 10 MBs. The configuration for how many logs to keep and the size of such logs can be seen in the .\agent\properties\glide.properties file. Example default values: java.util.logging.FileHandler.pattern=logs/agent%u.logjava.util.logging.FileHandler.append=truejava.util.logging.FileHandler.limit=10000000java.util.logging.FileHandler.count=10 The MID server logs can be retrieved directly on the instance. To collect the logs via the instance: Navigate to the MID server record of interest, MID Server > ServersClick on link "Grab MID logs, files and thread dump" In the ecc_queue we can see output records for topic SystemCommand and source "grabLog"Related list "Agent Logs" and "Agent Files" will be populated with the uploaded files by the MID server Note: Above steps will not be available if the MID server status is "Down" Troubleshooting Basics Set MID server parameter mid.log.level to "debug"Set any additional debug properties or parameters accordinglyReproduce the issueReview the MID server logs The mid.log.level may not be the only necessary property to be set when debugging, since different ecc_queue probes will be processed by different code. However, it is a good starting point for when troubleshooting.