MID Server high CPU consumption


Description

The MID server is used by many Servicenow applications and integrations. Some of the applications are Discovery, Service Mapping, Orchestration, Event Management, as well as a multitude of integrations. At times, the MID server JVM may use up more CPU than expected in the host server. This KB highlights some of the reasons CPU could go high on a MID server and how to find out what thread in the JVM is using up the resources. Knowing what thread is using up the resources is important to finding out a root cause as well as what team would be best suited to further investigate the issue.

Note: The MID server at times will use up 100% of the CPU while the MID server is actually busy. This would usually only be considered a problem if happening for long periods of time and affecting server performance, or if using up resources when not doing any actual work.

Reasons For High CPU Usage

Antivirus

When the MID server is being used it will create files on the user's temporary folder and on the mid server folder itself. A MID server that is heavily utilized may create and remove many such files. Antivirus software may attempt to review such files and use up CPU resources while doing so.

As a workaround:

Note: If necessary, the temp folder can be updated via parameter java.io.tmpdir. See KB0747569 on to change the temporary directory.

Powershell

When windows discovery is run it will create Powershell processes. When running lots of windows discovery these can add up and lead to high CPU usage. 

As a workaround:

Note: As outlined on our mid server system requirements documentation "you can set threads.max as high as 200, however, this setting may need to be changed depending on the OS. For example, Windows-based MID Servers running Discovery uses Powershell which is resource intensive. In this case, you may exhaust the CPU power of the host machine with just 50 threads count." However, depending on the CPU this could happen at even lower thread counts and thus the max thread count may need to stay under 50, or even lower than the default of 25.

Find High CPU Thread

For windows MID servers:

  1. Get a thread dump of the MID server per https://support.servicenow.com/kb_view.do?sysparm_article=KB0656229
  2. Install process explorer from https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx
  3. Open Process Explorer and find the process with high CPU usage
  4. Righ click on the process and select "Properties"
  5. Click on the "Threads" tab
  6. Order the threads by CPU usage and get the TID using up the CPU
    Threads
  7. The TID on the previous step is in decimal. The thread dump has TIDs in hexadecimal. Convert the TID to hexadecimal using any appropriate tool. The default calculator for windows can be used
    Decimal To Hexadecimal
  8. Open up the wrapper.log and find the thread
    Wrapper Log

Additional Information