High CPU on target CI when running service mapping discovery - Support and Troubleshooting

High CPU on target CI when running service mapping discovery Description High CPU on target CI when running service mapping discovery.

When Service Mapping (vertical discovery) runs a discovery it will create tasks. These tasks are sent to the MID servers via the ecc_queue. The first tasks will be created according to the entry points configured in the application service. Each task will run a pattern on the target IP address to collect data. Such task may return with information which will lead to a connection, created to endpoints, and therefore new tasks. This process repeats until there are no more connections to be created or tasks to be processed.

The tasks will be spread among the MID servers according to the MID MID Server Selection algorithm.

The IP address seen in the ecc_queue in most cases will be the "main" ip for the CI. The "main" ip is the IP set in the ip_address field for the CI. For example:

A connection is created for IP B which belongs to a windows server as seen in the cmdb_ci_ip_address table However the cmdb_ci_win_server.ip_address is A Therefore in the ecc_queue we would see an output record for IP A. Note : If the IP is found in the load balancer IP Service table (cmdb_ci_lb_service) then an entry is created for the load balancer.

Cause In general, High CPU on a target server could be seen due to, but not limited to, the following:

Too many tasks are sent to a target server, often happens when: A step in the pattern creating unnecessary connections. Review the pattern log to determine if this is the case. Traffic-based discovery runs A pattern has a step which may be CPU intensive Target server does not have enough resources allocated Resolution Too many tasks are sent to a target:

Pattern logic creating multiple connections Review pattern log to determine why so many connections are being created and adjust the pattern Traffic-based discovery runs Determine if a connection was created by Traffic-based discovery Disable Traffic-based discovery Additionally, system property sa.max_concurrent_service_discovery_tasks and MID server parameter mid.servicewatch.max_concurrent_connections can be adjusted to limit how many tasks can be executed concurrently. Lowering the number of concurrent tasks running against a target should lower resource consumption on the target as well.

System property sa.max_concurrent_service_discovery_tasks verifies that the number of concurrent non inclusion Service Discovery tasks does not exceed certain limit, if so wait before creating more tasks. The Default is 100.

MID parameter mid.servicewatch.max_concurrent_connections limits the number of concurrent tasks sent to an individual host by a single MID. The default is 7.

Finally, there could be too many MID servers configured to connect to a target. If that is the case, configuration changes may be needed to control how many MID servers can run tasks against a target. This could be accomplished by updating the IP ranges on the MID servers or the applications setup for such MID servers.

Disable Traffic-based discovery To turn off traffic based discovery:

Full turn off of traffic based discovery Set system property sa.traffic_based_discovery.active = false Turn off for specific CI or CI type Create a traffic based rule on table sa_traffic_based_rules For more information see Enable traffic-based discovery for CI types or specific CIs Turn off for specific service Navigate to the application service page and set "Traffics base discovery" = false (Note: This last option will stop the creation of traffic based connections in the map. However, it will still create tasks to run on the target server in the background. Therefore resources will still be utilized on the target and this option will not help alleviate resource consumption on the target). Determine if task output was created by Traffic-based discovery Open ecc_queue record On the payload look for the endpoint sys_id, for example: Navigate to cmdb_rel_ci and find relationship where: CHILD = CI with sys_id found in ecc_queue payload TYPE = "Use End Point To::Use End Point From" Copy the sys_id of the relationship Navigate to table cmdb_rel_attributes Search for record where relationship field is the same as the sys_id of the relationship If the "Qualifier Type" column is "Traffic Based" then the output was created due to traffic based discovery Find connections created by the pattern Open the service map pattern discovery log for CI Review the connection sections Steps which have created connections will have "[created]" added to the step name