MID Server Sizing for Discovery - Support and Troubleshooting

MID Server Sizing for Discovery .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Table of Contents Overview Summary System Requirements Scaling Strategy When to Scale Horizontally (Add More MID Servers) Load Distribution Approaches Factors Affecting Load Configuration and Tuning JVM Configuration Thread Pool Tuning CPU Usage Expectations Monitoring Best Practices Deployment Capacity Planning Discovery Optimization Troubleshooting Windows Discovery Performance Issues General Troubleshooting Configuration Checklist Initial Deployment Windows Discovery Ongoing Optimization Troubleshooting Key Principles Additional Resources Overview This article provides guidance on sizing MID Server deployments for Discovery operations, covering hardware requirements, scaling strategies, and Windows Discovery specifics.

The goal is to provide information to help make decisions - it is not a strict set of recommendations and it should not be considered as such.

Please also note that MID Server sizing is only one factor in overall Discovery performance - Instance performance must also be considered and it is not always appropriate to scale out the MID deployment as it could cause or increase instance-side performance concerns.

Before implementing any recommendations from this article, it is recommended to be familiar with the MID Server Fundamentals .

Summary Dedicated MIDs and clustering:

This article assumes MID Servers used for Discovery perform only Discovery activities Other applications such as Event Management, LDAP Sync, Import/Export, Agent Client Collector etc., will all have different performance requirements - if they also run on the same MID it can become difficult to estimate Performance requirements If particular Discovery Schedules are observed to have high performance requirements, consider creating a dedicated MID Server Load Balancing Cluster Likewise, if particular Discovery Schedules have strict networking requirements or MID Server configurations required, consider creating dedicated MID Server Load Balancing or Failover clusters Most of this information is relevant primarily to Windows MID Servers and for discovering Windows target devices MID Server thread pools:

Interactive Configurable with "threads.interactive.max" parameter - default: 10 This is used for the highest priority jobs, such as MID heartbeat, Pattern Debugger or "Test Credential" activities Generally to be used for jobs that should return information to the instance as soon as possible or which must take precedence over other jobs Expedited Configurable with "threads.expedited.max" parameter - default: 20 This is commonly used for medium-high priority jobs, such as "Discover Now"/"Quick Discovery", or REST Message jobs Generally to be used for jobs that are more important than "standard" but that do not need to run Standard Configurable with "threads.max" parameter - default: 25 This is where most of the MID's scheduled work will happen Scheduled Discovery will run in the Standard threadpool, and therefore this is the main threadpool to consider for right-sizing MID Servers for Discovery PowerShell Execution:

PowerShell processes run on the host OS, not within the MID Server application This means that the memory consumed by PowerShell processes is not part of the MID Server's allocated maximum heap size Choose between WMI and WinRM protocol with "mid.windows.management_protocol" parameter Consider the compatibility within your environment: WinRM requires that WinRM is enabled on the target servers, which may cause compatibility concerns for target servers with older Operating Systems WinRM is more performant than WMI WinRM has additional configuration options available such as specifying PS Session options or using Microsoft JEA Default limit: 25 concurrent PowerShell sessions Configurable with "mid.powershell_api.session_pool.max_size" parameter Note that threads from any pool may use a PowerShell session, and therefore this should be scoped to account for expected workload across all threads It is generally advisable to keep this approximately in line with the standard worker thread configuration Memory Considerations:

Review System Requirements section for recommendations JVM heap beyond 4 GB provides diminishing returns for MIDs dedicated to Discovery Only really beneficial if increasing threads beyond common configurations Patterns that produce large payloads - such as Kubernetes, Cloud Discovery or certain application patterns may benefit from memory configurations beyond this if executing many at once Consider allocating host RAM generously - as PowerShell processes run outside the MID Server JVM they will not be bound by the JVM heap size and may consume more memory than expected Each PowerShell process is separate and must load the standard PowerShell profile and all the relevant MID Server PowerShell Modules required for discovery A simple recommendation is to allocate enough host RAM to run the host OS (measured before installing the MID) + 2x the Mid maximum JVM. Note: this is a generous estimate and should be reviewed during heavy MID use to ensure that it is not overprovisioned This is a good rule of thumb when considering multiple MIDs on a single host Windows server, but does not scale well with single large MID installations For example - if a Windows server consumes 8GB in normal background processes and there are 2 MIDs installed each with max JVM memory of 2GB, a total allocation of 16GB to the host would likely be sufficient. CPU Considerations:

Review System Requirements section for recommendations A minimum of 4 CPUs at 2.0GHz each is required for the MID Server application The MID Server is designed to complete all its assigned tasks as quickly as possible, and will commonly run at high CPU usage while actively processing tasks CPU requirements vary drastically based on assigned workload An idle MID may consume very little CPU A standard MID with minimal requirements may likely hit 100% CPU usage when consuming all threads for Discovery A MID with generous system resource allocation may still hit 100% CPU usage during some discovery schedules, depending on the patterns or probes executing As with memory, PowerShell processes run outside the JVM and are a common source of high CPU usage CPU allocation should be monitored and adjusted A generic recommendation is to start with one CPU core per 6 expected concurrent threads for light usage, or one CPU core per 3 expected concurrent threads for medium-heavy usage If you configure your MID Server with the standard 25 threads and expect occasional usage for intermittent scheduled discovery (a few times per day), start with the minimum 4 CPU cores If you configure your MID Server with the standard 25 threads and expect continual usage for constant or frequent scheduled discovery, consider starting with 8 CPU cores Adjust from here in line with your observations Monitoring:

Consider adding exceptions on monitoring tools for MID Server CPU Usage - it is expected for the MID to burst to high cpu usage to complete discovery MID Server thread activity is visible in "Statistics (3 hours)" related list on MID Server form Consider dedicated monitoring tools, or utilise ServiceNow features like MID Server Resource Threshold alerts System Requirements MID Server System Requirements vary depending on what you are trying to achieve. MID Servers can be dedicated to a single application (like Discovery) or shared across multiple applications (such as Discovery + IntegrationHub, or Discovery + LDAP Sync).

This KB discusses recommended system specifications for dedicated Discovery MID Servers. Additional, or different applications have different requirements and would require additional resources.

The below table shows sample MID Server sizing recommendations based on the size of environment being discovered.

Environment Size Devices MID Servers CPU cores Host RAM (GB) MID JVM Memory (GB) Notes Small 1-2 4 8 2 Low volume of Discovery Medium (Mixed) 1000-5000 2-4 4-8 8 2 Environment containing mix of devices Medium (Windows) 1000-5000 2-4 8 12-16 4 Environment with mostly Windows devices Large 5000-15000 4-8 8 16 4 Consider acceptable Discovery runtimes Enterprise 15000+ 8+ 8+ 16+ 4+ Consider location-based MID clusters

Please review the MID Server System Requirements documentation page for further information including sample case studies

Scaling Strategy When to Scale Horizontally (Add More MID Servers) Discovery schedules do not complete in preferred time Discovery schedule for network segment does not complete within defined maintenance window Unable to discover each target device at preferred rate (for example, daily) Geographic distribution needed Network segmentation or security posture requires dedicated MIDs e.g., in DMZ High availability required CPU usage pinned at or close to 100% for excessive periods of time Performance-related problems causing errors and exceptions during Discovery Please note, CPU usage is expected to reach 100% during Discovery - the MID aims to complete the Discovery probes as quickly as possible. This is not inherently concerning, but continuous time spent at 100% CPU may cause problems and introduce performance-related problems.

A general recommendation is to deploy 3-5 medium sized MID Servers over 1 powerful MID Server to avoid performance-related problems or redundancy issues.

Examples of performance related problems caused by high CPU usage could include CPU backpressure or co-stop, which may prevent PowerShell executing successfully. See troubleshooting section below for more information.

MID Server Resource Threshold Alerts can help determine when each MID is overutilized.

Load Distribution Approaches Geographic: Per data center/region Functional: Separate Discovery, Orchestration, Integrations Network Segment: Per subnet/VLAN Capability-Based: Windows, Unix/Linux, network devices Please note: a Windows MID Server is required to discover Windows devices, but other discovery (network devices, Cloud discovery) is fully supported on Linux Server-hosted MIDs.

Consider the above factors if choosing to deploy multiple sets of MIDs - a good approach is to use MID Server clusters and link the appropriate clusters to each Discovery schedule or use Discovery behaviours.

Factors Affecting Load Target device count and complexity Discovery frequency and concurrency Time-based requirements If you must fit Discovery within a maintenance window, consider scaling out a dedicated cluster of MIDs per network section/window Likewise, if there are no strict timelines it may make sense to reduce the deployment of MIDs for a slower rate of Discovery Pattern complexity Certain Patterns may produce larger payloads or require higher System Resources (CPU, JVM/Host memory) to execute Network latency Credential volume Windows: PowerShell session count, admin$ accessibility, WMI efficiency Please note: MID Servers are not the only factor in Discovery performance. MIDs are traditionally the bottleneck for discovery performance but a highly scaled-out MID deployment may return information to the instance faster than it can be processed.

Consider your Instance performance when scaling out discovery/MIDs - it can be more effective overall to artificially throttle Discovery by decreasing the MID deployment to prevent overwhelming the instance's scheduled workers.

Additionally, consider implementing MID Server Resource Reservation to ensure that MIDs do not attempt to execute too many large tasks at once - this can be an effective way of preventing performance issues without simply scaling up or out the MIDs.

Configuration and Tuning JVM Configuration Default JVM heap: 1 GB (1024 MB) Windows Discovery recommended: 2-4 GB Large Patterns: 4+ GB based on needs

File: /agent/conf/wrapper-override.conf

Example:

wrapper.java.maxmemory=4096 Key Principle: No benefit beyond 4 GB for Windows Discovery. Allocate remaining RAM to host OS for PowerShell processes.

Thread Pool Tuning Default Standard Worker threads: 25 (configured in config.xml as threads.max parameter)

Start with defaults Monitor utilization in MID logs and Statistics Consider increasing if Discovery is too slow If performing lots of Windows Discovery: Balance with PowerShell session limit (default: 25) If a standard worker thread has to wait a long time for a PowerShell session, then simply increasing the thread count does not solve the underlying problem Note: More threads = higher CPU usage; find optimal value for your environment Consider using MID Server Resource Reservation to control the CPU usage instead of increasing if time to complete Discovery is not a factor Consider using MID Server Resource Threshold Alerts to understand when the MID Server is overutilized CPU Usage Expectations 100% CPU during discovery is normal and intended ( ServiceNow KB0952290 ). MID Servers maximize resources to minimize discovery duration.

Despite expecting 100% CPU usage when the MID is busy, this can still cause issues - occasional bursts to 100% are an effective way to complete Discovery quickly but excessive time at 100% CPU can cause OS-level performance issues that cause Discovery to fail or may cause stability issues for the host OS.

Abnormal Patterns:

CPU pinned at 100% continuously (not temporary spikes) CPU Blocking or Co-Stop experienced CPU Backpressure High CPU outside scheduled windows System unresponsiveness Infrequently Used MID Servers should be reviewed to ensure resources are not wasted:

Consider consolidating with other MID Servers Implement MID Server Resource Reservation or reduce thread count to throttle performance Monitoring CPU: Target Note: for less frequent discovery, aim for Memory: Target Queue Depth: Should clear between discovery runs Discovery Duration: Track completion times and trends Consider reviewing Discovery Performance Metrics Locations to monitor or check MID Server performance:

MID Server Dashboard: MID Server > Dashboards > MID Server Health ECC Queue: MID Server > ECC Queue (input/output depth and age) Discovery Status: Discovery > Status > Discovery Status Statistics: "Statistics (3 hours)" related list on MID Server form (view MID Server thread usage and stats) Configure MID Server Resource Threshold Alerts Best Practices Deployment Separate Hosts for production and non-production MID Servers Deploy at least 2 MID Servers per critical environment for redundancy Position MID Servers close to target infrastructure Separate Discovery from Integration/Orchestration workloads Don't overprovision - prevent overwhelming the instance with discovery payloads Capacity Planning Plan for 12-18 months of infrastructure growth Maintain 30% headroom for load spikes Review performance metrics quarterly Start conservative, scale based on metrics Discovery Optimization Stagger discovery schedules to avoid concurrent peaks Use precise IP ranges and/or Discovery Behaviours Troubleshooting Windows Discovery Performance Issues Common Causes:

Antivirus Interference: AntiVirus may inspect PowerShell processes, increasing CPU resource usage or blocking execution Confirm with procmon - monitor the PowerShell processes - or check AV logs Add MID directory and PowerShell processes to AV exclusions PowerShell Profile Modifications: PowerShell profiles will be loaded at run-time, which may consume excessive memory ServiceNow-provided PS modules will be imported into the running session after environmental powershell profiles Reset to default profiles for MID service account Enhanced Auditing: PowerShell transcripts are a very useful tool for auditing and troubleshooting purposes Utilising PS transcripts may consume excessive disk space, or cpu resources Consider disabling PowerShell transcription logging for MID service account if the transcripts are not required for auditing purposes Consider MID Server Command Audit tool if more auditing information is required Insufficient CPU: Slow execution of discovery commands, patterns etc., may be a sign of insufficient CPU allocation Commonly, if there is not enough CPU available, MID Server logs may show "PowerConsole" busy errors Missing admin$ Access: Verify admin$ share and credential permissions, if this is missing Discovery may revert to legacy WMI behaviour which is slower and less efficient If using "mid.powershell.target_base_dir", ensure that the share is provisioned on the target server or it may revert to fallback logic Advanced Issues:

CPU Backpressure: Windows OS throttles MID to prevent resource starvation. Reduce concurrent PowerShell sessions or add CPU cores.

VM Co-Stop: Over-provisioned VMs (e.g., 20 vCPUs) cause scheduling delays. Start with 8 vCPUs, scale horizontally if needed. MID servers may also be victims of co-stop caused by other VMs on the same host hypervisor.

General Troubleshooting High CPU (Pinned at 100%):

Verify abnormal (not temporary spikes during discovery) Check for overlapping discovery schedules For Windows: Check AV, PowerShell profiles, auditing Add CPU cores or additional MID Server Optimize or disable problematic patterns Memory Exhaustion (OutOfMemoryError):

Increase JVM heap (do not increase endlessly, there are diminishing returns and this may indicate other issues such as memory leaks) Review custom scripts for memory leaks or search ServiceNow documentation for known errors Add host RAM (especially for Windows) Restart MID Server after changes Slow Discovery:

Test network connectivity/latency Increase thread counts incrementally Consider adding MID Servers ECC Queue Output Buildup:

Verify MID connectivity to instance Packet Loss can cause issues with ECC Sender, or ECC Monitor AMB Channel disconnection may slow MID's response to posted ECC queue outputs Check MID resource utilization Increase MID capacity if constrained ECC Queue Input Buildup:

Likely an indication of Instance-level performance concerns rather than a MID issue Consider throttling MIDs by applying lower thread counts Review for custom Business Rules, sensors, pre/post sensor scripts Configuration Checklist Initial Deployment [ ] Size host per device count guidance [ ] Configure JVM heap (2-4 GB recommended) [ ] Set up network connectivity and firewall rules [ ] Configure MID capabilities and IP ranges [ ] Test discovery against sample devices Windows Discovery [ ] Add AV exclusions for MID directory and PowerShell [ ] Verify default PowerShell profile for MID service account [ ] Consider if PowerShell transcription logging is required [ ] Verify admin$ share accessibility - or configure "mid.powershell.target_base_dir" to use a different location (note: must be a network share defined locally on the target server) Ongoing Optimization [ ] Review performance metrics monthly [ ] Monitor queue depth and discovery duration [ ] Tune thread counts based on utilization [ ] Plan capacity for infrastructure growth Troubleshooting [ ] Monitor thread usage and PowerShell session utilization in Statistics [ ] Review System Resource usage [ ] Check for CPU backpressure or co-stop indicators Key Principles Start Small, Scale Up/Out: Begin conservative, monitor metrics, increase based on need Scale host System Resources (CPU core count, RAM) and MID Server Configuration Parameters together Horizontal scaling is generally preferred to vertical scaling Expect 100% CPU: During discovery is normal and intended; only investigate if pinned continuously Environment-Specific: There is no universal formula; audit periodically and adjust Additional Resources MID Server System Requirements MID Server 100% CPU Usage During Discovery MID Server High CPU Consumption MID Server Thread Pools MID Server Parameters