MID Server runs out of memory due to memory leak. - Support and Troubleshooting

MID Server runs out of memory due to memory leak. Issue  Symptoms MID Server memory usage trends up over time eventually exhausting the configured heap size and run into OutOfMemory errors. Once memory leak is severe enough it might not be able to respond to heartbeat probes from the Service Now instance and the instance will mark the MID Server as down.

Release Jakarta and Newer.

Cause Heapdump analysis shows there are memory leak due to H2 Database connection leak. H2 is an in-memory Database used by Discover Patterns and Operational Intelligence. There is a bug where if an exception occur while executing a SQL query, the connection is not cleaned up and accumulates. Eventually this leads to memory exhaustion. Below is screenshot of the leaked H2 connection objects when performing Heap analysis:

Resolution This is fixed in PRB1293425 for Kingston Patch 8.

Additional Information In MID Server's agent0.log[0-9], look for the "LogStatusMonitor" log entries. These are MID Server health checks. If you look at 'used' memory, it trends up over time even when it's not processing any probes.

09/01/18 23:23:50 (616) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 496.0mb, used: 89.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/02/18 23:23:50 (616) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 712.0mb, used: 243.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/03/18 22:24:50 (327) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 910.0mb, used: 498.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/04/18 23:57:50 (567) LogStatusMonitor.60 stats threads: 87, memory max: 910.0mb, allocated: 910.0mb, used: 732.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/05/18 10:28:18 (491) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 910.0mb, used: 877.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes