How to remotely monitor per-process CPU and Memory usage of a MID Server host


Description

While debugging MID Server issues, it may be helpful to monitor the host server's performance metrics, such as CPU usage, continuously, while performing tests. This would normally require a remote desktop of the MID Server host, and using tools such as Task Manager or Performance Monitor. However this can also be done from the command line, and that means you can do this remotely via the instance, as if you were running a probe command.

Instructions

For Windows Servers, the typeperf command outputs the same performance data as Task Manager or Performance Monitor.
https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/typeperf

The Linux top command is equivalent.
http://man7.org/linux/man-pages/man1/top.1.html

These can be run on a MID Server host using a "Command" probe. How a "Command" topic ECC Queue output should be formed is documented here:
https://docs.servicenow.com/csh?topicname=ecc-queue-mid-server.html&version=latest#manage-ecc-queue-content

All available counters can be listed with this command:

typeperf -q

Example 1: The Host CPU and Memory

CPU, Memory in use, Memory available:

typeperf "\Processor(_total)\% Processor Time" "\Memory\Committed Bytes" "\Memory\Available Bytes" -si 10 -sc 6 

Output is CSV Format, lists CPU usage, for all running processes, 6 times (-sc), 10 seconds apart (-si). 

"(PDH-CSV 4.0)","\\XXX\Processor(_total)\% Processor Time","\\XXX\Memory\Committed Bytes","\\XXX\Memory\Available Bytes"
"08/03/2020 17:05:38.187","3.492081","26921250816.000000","5113982976.000000"
"08/03/2020 17:05:48.189","3.832699","26869493760.000000","5143330816.000000"
"08/03/2020 17:05:58.191","3.407628","26836049920.000000","5165826048.000000"
"08/03/2020 17:06:08.192","3.106549","26834079744.000000","5164392448.000000"
"08/03/2020 17:06:18.193","4.319378","26830151680.000000","5168476160.000000"
"08/03/2020 17:06:28.195","3.806544","26825379840.000000","5251788800.000000"

The result will need copying from the ecc queue input record payload to a text editor, where it can then be saved as a .csv file. That can then be opened in Excel or similar.

Example 2: CPU of every Process

typeperf "\Process(*)\% Processor Time" -si 10 -sc 6

Column labels will be the process names. Each row will be a snapshot at a specific timestamp. The above example would give 6 rows of data, 10 seconds apart. You could do something different, like 120 rows, 60 seconds apart, if you wanted to get a bigger picture.

The main MID Server service processes may appear named something like:

Processes launched by the MID Server for particular probes may look something like this: 

Note: These processes will be outside the java application, and so will be using their own CPU and memory. That memory usage will be in addition to the java heap space allocated to the MID Server.

Example 3: PID, CPU, Memory and Handles of every Process

typeperf "\Process(*)\ID Process" "\Process(*)\% Processor Time" "\Process(*)\Working Set" "\Process(*)\Working Set - Private" "\Process(*)\Handle Count"  -si 10 -sc 6

Unfortunately the output is in counter order, not process, so not ideal for reading.

You may want to narrow it down to particular processes. e.g. all the 'java' and 'powershell' ones:

typeperf "\Process(java*)\ID Process" "\Process(java*)\% Processor Time" "\Process(java*)\Working Set" "\Process(java*)\Working Set - Private" "\Process(java*)\Handle Count" "\Process(java*)\ID Process" "\Process(powershell*)\% Processor Time" "\Process(powershell*)\Working Set" "\Process(powershell*)\Working Set - Private" "\Process(powershell*)\Handle Count" -si 10 -sc 6

Note: Using 'Working set' and 'Working Set - Private' for seeing how much memory the process is using is not totally reliable, but is still useful.

Example 4: Putting it all together in a Scheduled Script in the instance

You could use this example script, customised for the MID Servers and Counters you are interested in, in a periodic Scheduled Script (sysauto_script):

// Scheduled script to run periodically, to create jobs for a set of MID Servers
// Uses a combination of examples 1 and 3 from this KB, but runs once for each job
// https://hi.service-now.com/kb_view.do?sysparm_article=KB0824725
// This will result in the counters for the host memory in general, and each running process, 
// in CSV format in the payload of the input in the ECC Queue
// v1 2020-08-05 David Piper

// -----------------------------------
// Setup:

// The command line. 
//Backslashes need escaping (\ to \\)
// be careful with quotes and apostrophes
var command = 'typeperf "\\Processor(_total)\\% Processor Time" "\\Memory\\Committed Bytes" "\\Memory\\Available Bytes" "\\Process(*)\\ID Process" "\\Process(*)\\% Processor Time" "\\Process(*)\\Working Set" "\\Process(*)\\Working Set - Private" "\\Process(*)\\Handle Count" -sc 1'; // Define which MID Servers get the job // You can define the filter on a list view of MID Servers, and then copy the query string into this variable.
// be careful with quotes and apostrophes
var encodedQuery = "name=ProdDiscoMid^status=Up"; // ----------------------------------- var midGr = new GlideRecord('ecc_agent'); midGr.addEncodedQuery(encodedQuery); // This limits which MID Servers are involved. midGr.query(); var eccGr = new GlideRecord('ecc_queue'); while(midGr.next()) { eccGr.initialize(); // new record eccGr.agent = 'mid.server.' + midGr.name; eccGr.name = command; // the command may be croped, so put it in the payload as well eccGr.queue = 'output'; eccGr.topic = 'Command'; eccGr.state = 'ready'; eccGr.priority = 1; //expedited, so discovery's 2 standard ones are not in the way eccGr.payload = '<parameters><parameter name="name" value="' + escapeXml(command) + '"/><parameter name="skip_sensor" value="true"/></parameters>'; eccGr.insert(); } function escapeXml(unsafe) { return unsafe.replace(/[<>&'"]/g, function (c) { switch (c) { case '<': return '&lt;'; case '>': return '&gt;'; case '&': return '&amp;'; case '\'': return '&apos;'; case '"': return '&quot;'; } }); }