MID Server Max Threads - Worker Groups - Priority and QueuesIssue <!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Table of Contents The MID Server and the ECC Queue support PriorityWhich features make use of Priorities The MID Server and the ECC Queue support Priority Higher priority jobs can jump the queue, and execute quicker, in higher priority threads: PriorityThread GroupNo. Threads (default)ParameterInput Queue in MemoryJava Prioritywork\monitors\ECCSender folder Examples of probes2 (default)Standard 25 threads.max500 (25*Threads)5output_2 JDBC, LDAP, Discovery...1 Expedited 20threads.expedited.max400 (20*Threads)8output_1REST, SOAP0 Interactive 10 threads.interactive.max40 (4*Threads)10output_0HeartbeatProbeSystemCommand: restartService, grabLog, etc. When the MID Server retrieves a batch of output records from the ECC Queue, it will prioritise the higher priority jobs first (lower priority field value). The MID Server will fetch more jobs than it has free threads if they are in the ecc_queue. These remain queued in MID Server memory until a thread is free. The ECC Queue output record will be in Processing state, but it may be some time before it actually starts to be processed in a thread. Note: The Processed timestamp field in the ECC Queue cannot be relied on for this reason, as all it really means is the time the MID Server retrieved the record, not when the job started to run. The higher priority jobs will be given more CPU time in the MID Server application once running. The thread pools map to different Java thread priorities, and Java will assign more resources to the job. Java is a multi-threaded platform, and will use all the CPUs and Threads available to it. A dedicated host server/VM with many dedicated CPUs/Cores as you can get your hands on is recommended, or threads may execute considerably slower at busy times than they might when the MID Server is almost idle. That is important if you are expecting "Interactive" jobs to complete quick enough to avoid user transactions waiting too long for forms to update. e.g. A single LDAPListener thread may process user updates a lot slower while Discovery is also running in all other threads. Priority also applies to the ECCSender thread when passing results back to the instance. This will process the temporary files in the output_0 folder first, then output_1, then output_2. e.g. If a large Standard priority JDBC Import set is mid-way through, an Expedited result will not have to wait for all the import rows to be sent back to the instance first. The output_s folder, for multiple sequential inputs from a single output, which includes JDBCProbe and LDAPProbe are procesed at the priority of the output record - TBC. In the case of Discovery and Orchestration probes, the priority of the ecc_queue input then dictates the priority of the sys_trigger scheduled job that processes the sensor. Most scheduled jobs and async business rules in the platform are priority 100, so interactive inputs would take priority over those. Discovery can be set to use System Events for sensor processing, instead of Scheduled jobs, which can help manage the load on the instance and prevent scheduler worker thread blockages. ECC Queue PriorityScheduler Worker Priority (default)System property2110glide.ecc.async.priority.standard1105glide.ecc.async.priority.expedited050glide.ecc.async.priority.interactive Which features make use of Priorities Generally everything should run at Standard priority, unless there is a very good reason why it needs to jump the queue. As of Vancouver, very few features deliberately set this to anything other than Standard, and those that do have often been as the solution (or cause) of a problem ticket. Discovery/Service Mapping will use the priority of the Discovery Status record for all probes. This may vary depending on how the Discovery is launched: Interactive: Pattern Designer debug mode, Cancel Discovery, XMLStats (PRB1564921)Expedited: Service Mapping, Discover Now, Quick Discovery (PRB1624010), Shazzam port scan probes of Standard priority schedules (PRB1408213)Standard: Scheduled Horizontal Discoveries, except Shazzam MID Server: Interactive: All SystemCommands, Test Credentials Orchestration: Expedited: Test Inputs in Activity Designer, Start workflow from the Workflow Editor, and now All normal workflow executions (PRB1301862) IntegrationHub Expedited: All IPaasActionProbeInteractive: RESTProbe for Refresh OAuth Tokens Cloud Management Standard: APIProxyProbes CAPI orchestrator Event Management Interactive: Pull Connectors, as new events need getting into the instance without delay Change Management Standard: Automatic and Manually triggered Discovery of Affected CIs LDAP Expedited: Test Connection and Browse (PRB1331240)Standard: Imports, Listener Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Up to and including Australia, and later Resolution<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } There is scope for certain real-time updated User-Interface triggered jobs, or quick integrations running as part of a user transaction, to be allowed to run as Expedited priority. Such jobs running at higher priority would improve the user experience, and prevent blocking instance threads while waiting for the response which would increase overall instance performance. A before insert business rule on the ecc_queue table is a simple way to bump up the priority of specific outputs. It is very important to include the specific Queue, State, Topic, Name, Source and Agent Correlator fields in the conditions to make sure you don't effect more jobs that you intend. However often those integrations would be better running truly asynchronously, with a Sensor dealing with the eventual response from a low priority job in the background, without holding up a user's form or instance thread at all. A MID Server dedicated to those jobs may be a better solution. If integrations really have to be as realtime as possible, then going via the MID Server is probably not the correct solution. A synchronous RESTMessageV2 API request, direct from the instance, implemented in a Script Include, is still the quickest method, if it can be done without impacting the instance performance. KB0694711 Outbound REST Web Services RESTMessageV2 and SOAPMessageV2 execute() vs executeAsync()Using Integration Hub is also not ideal in that situation, due to the Flow engine running via System Events and Scheduled Jobs, which can normally be expected to add delays, in addition to delays due to the round trip to the MID Server and back.KB1179826 Flow are getting queued or delayed execution If you intend to improve the throughput of your MID Server, then you do need to consider the number of threads, as well as the available CPUs. A lot of probes spend a lot of their time just waiting for responses from endpoints, blocking threads, without using CPU. If you choose to raise the Standard thread pool number - up to 200 is possible before hitting limits in Windows - you should also scale up the other thread pools in proportion as well. Don't only increase the number in threads.max, without also doing threads.expedited.max and threads.interactive.max. Don't forget to scale up the JVM heap memory, and any host memory for Powershell processes too. With only 3 priorities, and really only 2 usable priorities, it is common for several sets of high priority jobs to be impacting each other, especially when there has been a flood of jobs due to a batch update. In this situation, consider dedicating MID Servers to each job, so they are fully protected from other jobs impacting them. Multiple MID Servers on ther same host is allowed. Important notes: Once all Interactive threads are in use, critical MID Server system commands can be blocked, including HeartbeatProbe which is used by the instance to know if the MID Server is up, so it could be set as Down and become unusable. It's therefore a good idea to never use Interactive, and use Expedited for anything that must jump the Standard queue. Only if testing showed you had a need to jump the Expedited queue would you consider Interactive. Running higher priority Sensor scheduled jobs - either too many in a short time, or a few long running ones - have been known to cause performance issues for the instance. Discovery and Orchestration sensors at Interactive priority are priority 50 scheduled jobs, which is higher than most jobs and all async business rules, and equal to very important things like the text indexer and event management jobs. This can be dangerous, and any implementation above Standard priority needs to consider this. Regardless of priority, threads can be stuck waiting for Connections from the various Connection Pools. If all threads are running the same type of probe, then the limits might be hit. Check the MID Server Parameters and Properties for the maximum connections and connections per host for various protocols, to avoid that happening. Agent Client Collector generally only uses the ECC Queue and Worker threads for jobs like an on-demand host data collection, or a policy config update, where it immediately passes to jobs on to the Agents, so priority or number of Worker threads isn't really relevant. The ACC Websocket Endpoint extension has some thread pool properties of its own for the bulk of ACC related processing, which can be tuned and scaled depending on the use case. These are threadpools specifically for the following actions: sn_agent.mid.ws_keepalive_pool_sizeMonitoring.KeepAlive thread pool size. keepalive processing, where it processes keepalive payloads from agents. (resizable without restart). Also carries the inline (non-blocking) outbound send of schedule/config-sync messages.sn_agent.mid.ws.event_pool_sizeMonitoring.CheckResponse thread pool size. Check response processing, where it processes check response payloads from agents (resizable without restart)sn_agent.mid.ws_rhino_pool_sizeMonitoring.ScriptInclude thread pool size. Script include processing, where it processes anything that is sent to a target mid server script (named-script check results) And while talking about Agent Client Collector, this is also an Agent, similar in function to a MID Server. It can connect directly to the Instance, without needing a MID Server involved, and can run integrations via the ACC Spoke, or custom Checks. Agent Client Collector-based, rather than MID Sever-based integration design might be better for some requirements. And finally, "When everything is a priority, nothing is a priority", so don't be silly with this in custom implementations or you will defeat the purpose. Related Links<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } MID Server System Requirements Discovery MID Servers at 100% CPU - Good or Bad, Why and then what?