MID Server max thread count should be increased - Product Knowledge

MID Server max thread count should be increased  Product Success Playbook

MID Server max thread count should be tuned for optimal performance

A step-by-step guide to tune performance of MID Servers

Table of Contents

Summary

Goal of this Playbook

Audience

Problem Overview

Executive Summary

How this playbook can help you achieve business goals

How this playbook is structured

Problem Analysis

Upstream Causes

Downstream Consequences

Impact on Your Business

Engagement Questions

Remediation Plays

Summary

Play 1: Review your MID Configuration Parameters

Play 2: Analyze your MID Configuration Parameters

Play 3: Use the Calculator to estimate your MID Server needs

Play 4: Set or adjust the global threads.max parameter

Data Governance

Summary

Goal of this Playbook

The goal of the Playbook is to help optimize MID Server performance.

Details about this playbook

Author John (Johnny) Walker Date 12/06/2021 Addresses HSD # HSD0009884 Applicable ServiceNow Releases All Releases Time Required Approximately 1 hour (depending on your environment) Audience

ServiceNow Administrator or Discovery Administrator

ServiceNow Platform Architects and Owners

Problem Overview

Large organizations often struggle with completing a regular horizontal Discovery (or inventory) in a timely and predictable manner. It is the primary job of the MID Server to perform the work of Discovery Schedules. We want Discovery to take place during the scheduled times or windows which have been deemed as "off peak" or otherwise appropriate. Completing all ITOM Health / Visibility work as efficiently as possible is obviously the goal of any enterprise.

The MID Server application is a standard Java process constrained by well understood performance (environment) variables. One of these environment variables is the max.threads setting which controls how many tasks can be carried out by the MID Server in parallel. The more tasks which can be performed at once, the faster all the work is completed.

The baseline settings for the MID Server allow the instances to be stood up and may even work for some development or testing purposes but these OOB settings are rarely appropriate for the mission critical work of a Production MID Server.

Executive Summary

How this playbook can help you achieve business goals

Having properly tuned MID Servers will enable your entire CMDB to be updated more frequently and hopefully entirely within an approved "off peak" window. ITOM Visibility (Discovery) populates the CMDB which is a strategic necessity in today's IT landscape. Data within your CMDB provides a foundation for operation of multiple business units and products.

How this playbook is structured

This playbook contains four plays to help you evaluate your threads.max parameter(s) for your MID Servers. The 1st play is to show you how to locate the parameters. The 2nd play is analysis of the settings you may find. The 3rd play provides you the MID Server Discovery Calculator (or estimator), a lightweight spreadsheet to help roughly gauge your needs based your CMDB size and number of MID Servers. The 4th play is quick and easy means to raise the threads.max value across all your MID Servers. Lastly, a governance play reminds you of the MID Server Dashboard where metrics and information are already available to help you manage things.

Problem Analysis

Upstream Causes

The MID Server default setting for max.threads is 25. If you have made no changes to this setting, your MID Servers will have this value

Perhaps your Discovery Administrators were not aware of the impact this setting can have on performance or chose to leave it as the default setting

Perhaps new MID Servers have been added to your system and have not yet been tuned for optimal performance

Downstream Consequences

Data Consequence

Your CMDB records could be getting stale for lack of Discovery Schedule completion due to a lack of available threads to perform work concurrently

Service Mapping tasks could be taking longer to complete due to a lack of resources Operation Consequence

Having MID Servers which are not optimized for performance can lead to additional work for your operational team to keep your data complete and accurate Your Discovery Administrators may be continuing to lengthen the max allowable time on Discovery Schedules

Your Discovery Administrators may be adding additional MID Servers, attempting to compensate for a lack of resources on existing MID Servers

Your Discovery Administrators may have allotted additional memory (heap) to MID Servers only to find that the change in performance wasn't as expected

Having added additional MID Servers to compensate, your Administrators will spend additional time managing those servers App Consequence

Discovery Schedules could cancel for lack of resources to complete them on time

Discovery Schedules can take longer than they need to, pushing them outside of the off-peak times they are supposed to run within

Orchestrated business processes using the MID Server during peek Discovery times can be impacted by a lack of resources

Impact on Your Business

Performance tuning for your MID Servers has a direct impact on the ITOM Visibility suite, specifically Discovery Schedules.

ITOM Visibility Discovery Schedules will take longer to complete when the MID Server has less threads available to perform tasks concurrently.

MID Server host entitlements (for OS and Virtual Machines) You may be able to decrease the number of required MID Server host machines (physical or virtual) when they are properly tuned for performance. Engagement Questions:

Consider the answers to these questions:

Who manages your MID Servers?

How was planning for the size and number of your MID Servers conducted?

When was the last time your MID Server number, performance and distribution reviewed?

Remediation Plays

Summary

The table below lists and summarizes each of the remediation plays in the playbook. Details are included later.

Play Name

Review your MID Configuration Parameters

What this play is about

Review your MID Configuration Parameters Required tasks

Navigate to these in list view Analyze your MID Configuration Parameters

What this play is about

Analyze your MID Configuration Parameters Required tasks

Fix Plays

What this play is about

Use the calculator to estimate your MID Server needs Required tasks

Use a lightweight excel document to calculate Discovery performance, roughly Data Governance

What this play is about

Required tasks

Play 1 - Review your MID Configuration Parameters

What this Play is about

The MID Configuration Parameters are available to view and adjust through list view. This play will take you to the list of them.

Required tasks

Navigate to 'ecc_agent_config.list' in your filter navigator.

Add the filter condition as Parameter name IS threads.max and then Run the filter.

Example:

Create a Favorite using the context menu (hamburger icon) so that you can easily return to this location, if needed.

Continue to the next play to analyze these settings.

Play 2 - Analyze your MID Configuration Parameters

What this Play is about

Having navigated to and filtered for the appropriate parameters, we will now discuss them. This play assumes you've began with Play 1 analysis.

Required tasks

If you have loaded the list view from Play 1 and the list is empty, then all of your MID Servers are using the default number of maximum threads which is 25.

If you notice that at least one of the entries in your list view has the value "(empty)" under MID Server, this means you have your own global default maximum threads. This means that any MID Server within this instance which does not have its own entry in this list will default to the Value on this empty entry.

This is by design so that a large number of MID Servers within your environment can uniformly adhere to a single configurable value for threads.max. See Play 4 - Set or adjust the global threads.max parameter for information on how to use this configuration.

If you notice several entries here, it may be helpful to order the list by the value column to look at where they are set.

Any entries here where the value is less than or equal to 25 was either set on purpose to return the MID server to the default setting - for some reason. It may be that a particular MID Server was encountering an error or was otherwise involved in troubleshooting.

We will be advising you to increase the threads.max value for your system. We are aware you may have questions and/or concerns related to these performance settings. Please take the time to read the section titled 'MID Server Performance CPU Case Studies' at MID Server System Requirements .

Continue to Play 3 to learn about calculating your MID Server implementation in terms of size, number and horsepower.

Play 3 - Use the MID Server Calculator to estimate your MID Server needs

What this Play is about

We are providing a lightweight excel document to help with estimating the performance and number of MID Servers based on the counts of devices expected in Discovery.

Required tasks

Gather the counts of Computers, Network Devices and Printers expected in horizontal Discovery schedules. A rough estimate is OK.

Gather the count of how many MID Servers you already have assigned to the Discovery application.

Make note of the threads.max value you may already have, or that 25 is the default if you have not set this parameter. Download the MID Server Calculator and open it.

Complete the fields with your gathered information relating to the counts of devices and number of MID Servers.

Example: Take note of the Total estimated time to completion in either minutes or hours as you adjust the threads count. While these numbers are generalized using assumptions and/or averages, you can easily see that the time required to complete Discovery is strongly impacted by increasing the number of threads available to your MID Servers.

For more information regarding the factors this calculator may not account for, such as MID Server applications sharing host resources with other installed applications (including other MID applications) please take the time to read the entire MID Server System Requirements .

Play 4 - Set or adjust the global threads.max parameter

What this Play is about

In case studies we have shown that increasing the threads.max parameter from 25 to 50 does not immediately create CPU pressure on the host machine. See the reference information at the bottom of this playbook for more details.

Required tasks

Starting from the list view in play 1, if there is a entry where MID Server reads "(empty)", open that record. If not, click New.

On the new form, ensure the name for the parameter is threads.max.

Set or edit the value for this record to be 50. This is a generally safe value to increase to from the default.

Save or update this record. Any MID Servers should immediately pick up this value if there is not a more specific threads.max parameter for the individual MID Server.

For more information please refer to the documentation on MID Server System Requirements. There is performance case study information for those who wish to dive in to this further.

Data Governance

What this Play is about

This play reminds us that there's always a dashboard available to help govern your ServiceNow system.

Required tasks

For information on how to use the MID Server Dashboard, review the documentation here .

Navigate in your system to MID Server > Dashboard

Congratulations You have completed this Product Success Playbook.