Discovery IP Optimization - Support and Troubleshooting

Discovery IP Optimization Summary IP Optimization is the process used by pattern discovery to avoid repeatedly discovering the same CI in a single discovery scan.

Probes/Sensor When using probes, this process is achieved in the identification phase of discovery. The identification phase will run before deciding whether or not to trigger exploration of the device. At the identification phase, the sensor will call the Identification and Reconciliation Engine (IRE) to either create or update the CI. Once this is done, the sys_id of the device is returned. This sys_id is then used to check the discovery_device_history records related to the current discover status. Exploration will not be triggered if a match is found, a match found means this CI is being discovered, or was already discovered, using another IP address in the same discovery status. The discovery_device_history field last_state will be set to "Identified, ignored extra IP" for IP addresses for which the exploration phase is not triggered.

See Script Include: DiscoveryJSONIDSensor

Patterns Patterns have combined the identification and exploration phase. Therefore, the exploration phase cannot be skipped using information from the identification phase. Instead, starting with the classification input, IP Optimization will check on table discovery_device_duplicate_ips to determine if a CI is already being discovered or not. The sensor therefore will:

Check in the IP Address and/or Network Card table for an IP which is a match for the source of the input. If there is a single match, get the CI for the matching record. If more than one result is returned for an IP address the input will not be skipped Check if there is already another match for this CI and discovery status in the discovery_device_duplicate_ips table. If there is a match, set the discovery_device_duplicate_ips state to "Skip IP" or "Duplicate IP". Ecc queue inputs will remain in ready state. At this point, if there wasn't a match, create a discovery_device_duplicate_ips with state "Process IP" Note: When the discovery completes for a device ip, the BR "Discovery Device Duplicate IP" will be triggered and find the other inputs for this device which are waiting in ready state and set them to processed so that the discovery can complete. Also, the above is more of an overview of this process, instead of the detailed/complete logic.

IP Optimization Requirements glide.discovery.ip_based.active = false glide.discovery.device.duplicate.ip.optimization = true Network Card record's related to CI have their install_status field = installed IP Address record's related to CI have their install_status field = installed Note: Network Card and/or IP Address for a device must have a single match in the CMDB. For example, suppose you discover a device with IP = 192.168.1.10 and there are two such IP addresses/Network Adapter in the CMDB then this IP will be fully discovered.

Default values for the properties mentioned above:

Name Value glide.discovery.ip_based.active true glide.discovery.device.duplicate.ip.optimization true Note: Probe to Pattern Migration: Procedure for switching from probe-based Discovery to pattern-based Discover ( KB0694477 ) should be followed before migrating from probe based discovery to pattern based discovery. The migration process will set "glide.discovery.ip_based.active" = false, except for instances migrating TO a version pre NewYork Patch 5 due to PRB1368993.

Issues Devices may be discovered multiple times on the same discovery when IP Optimization fails. This may lead to a number of different issue, for example:

Node memory issues If the device is discovered multiple times in parallel, and its input payload is processed on the same node, this could cause memory contention on the node Replication lag This could be seen when discovering a very large switch or router multiple time on the same discovery. Such devices can bring thousands of records and updating all such records could cause replication lag Discovery cancelled or hangs, it could hang if the discovery does not have a max run time set Ecc_queue inputs may remain in ready state due to PRBs in the IP Optimization logic Troubleshooting Discovery hangs Open the discovery status Select the "ecc_queue" tab Filter for "processed" is empty and queue is input Make note of the IP addresses where processed is empty Navigate to discovery_device_duplicate_ips table Filter for the "source" using the IPs found in step 3 AND discovery status of the current discovery Show matching on "CMDB CI" and remove the source filter condition At this point, you should see all the IPs discovery found for the CI Check for errors in the ecc_queue inputs for the other IP addresses used for this CI Device discovered multiple times in the same discovery status The following could cause a CI to be discovered multiple times on the same discovery:

Duplicate ip addresses on table CMDB_CI_NETWORK_ADAPTER for the CI being discovered IP address discovered does not exist in either table: CMDB_CI_NETWORK_ADAPTER CMDB_CI_IP_ADDRESS System properties not set according to the "IP Optimization Requirements" section of this KB Network Switches & Routers Special Notes Network switches and routers have a greater potential of issues with the IP Optimization logic. Such devices may end up being discovered many times and cause unnecessary resource consumption. This is often due to:

Loopback addresses Network addresses Broadcast addresses Depending on the release:

Loopback addresses are not added to the network adapters table (they are filtered out because they are loopback addresses) Network and broadcast addresses are discovered Loopback addresses PRBs PRB1352376 and PRB1354567 added the option to discover loopback addresses. This is controlled by property glide.discovery.allow_loopback_adapters

Network & Broadcast addresses These will not be added to the NICs table, however they will not be discovered. PRB1358510 fix checks against network addresses and PRB1342008/PRB1394394 will skip broadcast addresses.

Determine if an IP is broadcast or network address If a device is being updated multiple times on the same discovery, and causing issues, we need to check if this is due to network and/or broadcast addresses being discovered.

Navigate to discovery_device_history Filter by discovery status (the discovery where the issue happened), cmdb_ci (the CI affected), and last_state not set to "Identified, ignored extra IP" Open some of the discovery_device_history records Navigate to the ECC Queue related list Open the input for the SNMP classification probe Get the source IP used by the probe on this discovery device history and search in the payload From the payload ipAdEntNetMask determine if this is a network or broadcast address Example ( Replace A and B with actual IP addresses ):

IPs A and B both discovered and updated CI in the same discovery status Open the classification input and search for both, following found: A 255.255.255.240 B 255.255.255.240 Using a subnet calculator, we find the following: Address: A Netmask: 255.255.255.240 = 28 11111111.11111111.11111111.1111 0000 Broadcast: B HostMin: NOT A or B HostMax: NOT A or B Hosts/Net: 14 (Private Internet) From the above, we see that IP address B is a broadcast address and therefore should not have been discovered. IP address A is neither a network or broadcast address and therefore this discovery should be ok. Solution (Network Switch & Router) Upgrade to Orlando patch six where both PRB1354567 and PRB1394394, loopback and broadcast. Set system property glide.discovery.allow_loopback_adapters = true Network addresses will continue being discovered until fix targeted in Quebec where PRB1436135 is fixed. Problems\KBs PRB\KB Description Fix Target PRB1319717 discovery_device_history records are deleted for devices with multiple IPs. As a result, the Discovery schedule fails to complete during the execution. LP8, MP3, NY PRB1358510 Horizontal Discovery does not detect Extra IP address in the same discovery schedules PRB1368993 Glide.discovery.ip_based.active not to be updated during probe to pattern migration NYP5 PRB1370895 Duplicate IP in schedule causes ecc_queue inputs to stay in ready state indefinitely and thus the discovery hangs until eventually cancelled PRB1334206 Error on "INSERT INTO discovery_device_duplicate_ips" causes ECC Queue Input record to stay on "ready" state and Discovery hangs MP10, NYP5, O PRB1401942 IP Optimization does not set ecc_queue inputs to processed if there is an error processing the input from the "main" ip and discovery hangs. PRB1483121 Discovery schedules with multiple MID Servers and multiple Shazzams can trigger several of the same CIM - Classify probes against the same CI PRB1394394 Discovery not running on the loopback address when netmask is 32 PRB1342008 IP Address field for switches are changed several times when broadcast ip_addresses ( for the customer network) are included in the Discovery schedules and we run patterns PRB1436135 Issue discovering a CI when scanning multiple IP's of same CI in single schedule where the first IP returns as being invalid IP (Network, Broadcast, HSRP VIP) PRB1387976 Discovery device duplicate IP's feature fails when the first IP fails to classify but the second IP that should classify gets marked as duplicate PRB1433712 SNMP classification doesn't detect broadcast\network addresses properly PRB1448131 'Unique Key violation' and 'QLIntegrityConstraintViolationException' errors on the discovery_device_duplicate_ips table in Paris PRB1444252 Discovery hangs on Paris caused by regressions of PRB1387976 PRB1503797 Discovery may hang if the classify input is larger than 512kb Quebec Patch 6 PRB1532619 Discovery does not detect an extra IP address in the same discovery schedules, this is causing Replication lag PRB1490404 A Linux server incorrectly parses bonded network interface controller (NIC) names, resulting in them being filtered out