MID Server Max Threads - Worker Groups - Priority and Queues - Support and Troubleshooting

MID Server Max Threads - Worker Groups - Priority and Queues Description Table of Contents The MID Server and the ECC Queue support Priority Which features make use of Priorities Making use of this information The MID Server and the ECC Queue support Priority Higher priority jobs can jump the queue, and execute quicker, in higher priority threads:

Priority Thread Group No. Threads (default) Parameter Input Queue in Memory Java Priority work\monitors\ECCSender folder Examples of probes 2 (default) Standard 25 threads.max 500 (25*Threads) 5 output_2 JDBC, LDAP, Discovery... 1 Expedited 20 threads.expedited.max 400 (20*Threads) 8 output_1 REST, SOAP 0 Interactive 10 threads.interactive.max 40 (4*Threads) 10 output_0 HeartbeatProbe SystemCommand: restartService, grabLog, etc. When the MID Server retrieves a batch of output records from the ECC Queue, it will prioritise the higher priority jobs first (lower priority field value). The same applies to the ECCSender thread when passing results back to the instance. e.g. If a large Standard priority JDBC Import set is mid-way through, an Expedited result will not have to wait for all the import rows to be sent back to the instance first.

The MID Server will fetch more jobs than it has free threads if they are in the ecc_queue. These remain queued in MID Server memory until a thread is free. The ECC Queue output record will be in Processing state, but it may be some time before it actually starts to be processed in a thread. Note: The Processed timestamp field in the ECC Queue cannot be relied on for this reason, as all it really means is the time the MID Server retrieved the record, not when the job started to run.

The higher priority jobs will be given more CPU time in the MID Server application once running. The thread pools map to different Java thread priorities, and Java will assign more resources to the job.

Java is a multi-threaded platform, and will use all the CPUs and Threads available to it. A dedicated host server/VM with many dedicated CPUs/Cores as you can get your hands on is recommended, or threads may execute considerably slower at busy times than they might when the MID Server is almost idle. That is important if you are expecting "Interactive" jobs to complete quick enough to avoid user transactions waiting too long for forms to update. e.g. A single LDAPListener thread may process user updates a lot slower while Discovery is also running in all other threads.

In the case of Discovery and Orchestration probes, the priority of the ecc_queue input then dictates the priority of the sys_trigger scheduled job that processes the sensor. Most scheduled jobs and async business rules in the platform are priority 100, so interactive inputs would take priority over those.

ECC Queue Priority Scheduler Worker Priority (default) System property 2 110 glide.ecc.async.priority.standard 1 105 glide.ecc.async.priority.expedited 0 50 glide.ecc.async.priority.interactive Which features make use of Priorities Generally everything should run at Standard priority, unless there is a very good reason why it needs to jump the queue.

As of Vancouver, very few features deliberately set this to anything other than Standard, and those that do have often been as the solution (or cause) of a problem ticket.

Discovery/Service Mapping will use the priority of the Discovery Status record for all probes. This may vary depending on how the Discovery is launched: Interactive: Pattern Designer debug mode, Cancel Discovery, XMLStats (PRB1564921) Expedited: Service Mapping, Discover Now, Quick Discovery (PRB1624010), Shazzam port scan probes of Standard priority schedules (PRB1408213) Standard: Scheduled Horizontal Discoveries, except Shazzam MID Server: Interactive: All SystemCommands, Test Credentials Orchestration: Expedited: Test Inputs in Activity Designer, Start workflow from the Workflow Editor, and now All normal workflow executions (PRB1301862) IntegrationHub Expedited: All IPaasActionProbe Cloud Management Standard: APIProxyProbes CAPI orchestrator Event Management Interactive: Pull Connectors, as new events need getting into the instance without delay Change Management Standard: Automatic and Manually triggered Discovery of Affected CIs LDAP Expedited: Test Connection and Browse (PRB1331240) Standard: Imports, Listener Making use of this information There is scope for certain real-time updated User-Interface triggered jobs, or quick integrations running as part of a user transaction, to be allowed to run as Expedited priority. Such as job running at higher priority would improve the user experience, and prevent blocking instance threads while waiting for the response which would increase overall instance performance. However often those integrations would be better running truly asynchronously, with a Sensor dealing with the eventual response from a low priority job in the background, without holding up a user's form or instance thread at all. A MID Server dedicated to those jobs may be a better solution.

A before insert business rule on the ecc_queue table is a simple way to bump up the priority of specific outputs. It is very important to include the specific Queue, State, Topic, Name, Source and Agent Correlator fields in the conditions to make sure you don't effect more jobs that you intend.

If you intend to improve the throughput of your MID Server, then you do need to consider the number of threads, as well as the available CPUs. A lot of probes spend a lot of their time just waiting for responses from endpoints, blocking threads, without using CPU. If you choose to raise the Standard thread pool number - up to 200 is possible before hitting limits in Windows - you should also scale up the other thread pools in proportion as well. Don't only increase the number in threads.max, without also doing threads.expedited.max and threads.interactive.max.

With only 3 priorities, and really only 2 usable priorities, it is common for several sets of high priority jobs to be impacting each other, especially when there has been a flood of jobs due to a batch update. In this situation, consider dedicated MID Servers to each job, so they are fully protected from other jobs impacting them.

Important notes : Once all Interactive threads are in use, critical MID Server system commands can be blocked, including HeartbeatProbe which is used by the instance to know if the MID Server is up, so it could be set as Down and become unusable. It's therefore a good idea to never use Interactive, and use Expedited for anything that must jump the Standard queue. Only if testing showed you had a need to jump the Expedited queue would you consider Interactive.

Running higher priority Sensor scheduled jobs - either too many in a short time, or a few long running ones - have been known to cause performance issues for the instance. Discovery and Orchestration sensors at Interactive priority are priority 50 scheduled jobs, which is higher than most jobs and all async business rules, and equal to very important things like the text indexer and event management jobs. This can be dangerous, and any implementation above Standard priority needs to consider this.

And finally, " When everything is a priority, nothing is a priority ", so don't be silly with this in custom implementations or you will defeat the purpose.

Additional Information MID Server System Requirements

Discovery MID Servers at 100% CPU - Good or Bad, Why and then what?

MID Server Max Threads - Worker Groups - Priority and Queues