Troubleshooting Shazzam Phase - Support and Troubleshooting

Troubleshooting Shazzam Phase Description This knowledge article covers how to Troubleshoot the Shazzam phase in Discovery and the solution to issues seen in the Shazzam phase.

Discovery reacts to the Shazzam port scan as follows:

If Discovery finds an IP address with a state of OPEN for any port communicating over WMI, SSH, or SNMP, Discovery lists the IP address in the Shazzam returns (result="open"). If Discovery does not receive a response from a port, it does not list the IP address in the returns from Shazzam. If Discovery receives a response from the scan that refuses the connection, Discovery lists the IP address and the result (result="refused") in the Shazzam returns. Symptoms Cannot discover some devices Cannot discover any particular type of devices (SNMP/SSH/WMI) Cannot discover some SNMP devices Cannot discover some LPS devices Is not turning information for a certain network segment, or for the entire network Error TCP_CONNECTION_FAILURE java.net.SocketException: No buffer space available Does not discovers the same number of devices consistently Shazzam takes a long time to complete Video Tutorial: Troubleshooting a failed Discovery: Scanning Phase

Resolution Shazzam cannot discover some devices It could be a network connectivity issue. Please check that the ports used by discovery as listed on Shazzam probe, port probes, and protocols documentation for the device type are not blocked.

Shazzam cannot discover any particular type of devices (SNMP/SSH/WMI) If it is not a network issue, first check the port_probe_spec Shazzam output payload and make sure your port is being scanned. If it is not scanned, then check the behavior of the schedule. For more information, see the documentation on Discovery Behaviors .

Also, it is possible that the port probes are not prioritized property. For more information, see the documentation pages on Selective Port Probe Scanning and Port Probes .

Shazzam cannot discover some SNMP devices If the solutions above do not work, credentials for SNMP devices are needed for detecting SNMP are alive. Add the necessary SNMP credentials for the device.

If the credentials for the SNMP device are already added and the device is still not discovered, check what snmp version the discovery schedule is set to discover via field "Use SNMP Version" (The default is v1/v2). Set "Use SNMP Version" to "All" in order to attempt snmp v1, v2, and v3.

Note: Out of box Shazzam will attempt community string "public". This can be the default for some SNMP devices and thus such devices may be discovered successfully even when there are no SNMP credentials under discovery_credentials. However, for most SNMP devices, snmp credentials need to be added to discovery_credentials in order to discover them successfully.

Shazzam is not discovering some LPS devices Make sure the device type port is open and the service is up and running on that device (for example, WMI service, SSH server).

Shazzam has probe parameters that may need to be fine tuned. For example, if Shazzam is not finding SSH devices, the GenericTCP_waitForConnectMS and BannerTCP_waitForConnectMS wait time may need to be extended. For more information, see the Configure Shazzam probe parameters and payload size documentation page.

Reducing the shazzam_chunk_size from 100 to a lower number may also help.

The MID Server is not turning information for a certain network segment, or for the entire network Check network connectivity using PING, TRACEROUTE, TELNET, and SNMP scanning tools.

Make sure the MID Servers that Discovery is using can reach the desired segments of the network. See the page: Configure an IP address range for the MID Server .

Error TCP_CONNECTION_FAILURE java.net.SocketException: No buffer space available (maximum connections reached?) Possibly due to Multi-thread Shaazam. See section "Single Thread vs Multi-thread Shazzam".

Does not discovers the same number of devices consistently With multi-thread Shazzam there will be an increase in the number of port scans at a specific time. Some network security devices and appliances may find this malicious and block/throttle port scan. This may lead to less IPs detected. See section "Single Thread vs Multi-thread Shazzam".

Shazzam takes a long time to complete on some MID servers This can happen if the mid server keystore is no longer valid. If the keystore is no longer valid the MID server will not be able to collect the credentials from the instance. This causes the SNMP scanner in Shazzam to attempt a new credential load for each ip discovered, which is an expensive operation. This also locks the other scanner threads and causes Shazzam to take a lot longer. The following can be seen in the agent logs when this happens:

ConcurrentPortScannerEngine-3: SEVERE *** ERROR *** MID Server encryption keys do not match and are no longer valid. To restore proper functionality, invalidate and re-validate the MID Server. ConcurrentPortScannerEngine-3: SEVERE *** ERROR *** An error occurred while decrypting credentials from instance com.snc.automation_common.integration.exceptions.AutomationIOException: No key pair found for DefaultSecurityKeyPairHandle at com.service_now.mid.keypairs.provider.standard.StandardKeyPairsProvider.decrypt(StandardKeyPairsProvider.java:182) at com.glide.util.MIDServerInfoPayloadDecrypter.decryptPayload(MIDServerInfoPayloadDecrypter.java:30) at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.loadCredentials(StandardCredentialsProvider.java:317) at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.load(StandardCredentialsProvider.java:287) at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.loadIfNecessary(StandardCredentialsProvider.java:299) at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.iterator(StandardCredentialsProvider.java:171) at com.service_now.mid.probe.shazzam.scanners.SNMP.getCredsIterator(SNMP.java:383) at com.service_now.mid.probe.shazzam.scanners.SNMP.sendSpamTaps(SNMP.java:283) at com.service_now.mid.probe.shazzam.scanners.SNMP.sendFirstTap(SNMP.java:252) at com.service_now.mid.probe.shazzam.scanners.SNMP.nextPhase(SNMP.java:199) at com.service_now.mid.probe.shazzam.PortScannerEngine.run(PortScannerEngine.java:70) at com.service_now.mid.probe.shazzam.ConcurrentPortScannerEngine.run(ConcurrentPortScannerEngine.java:35) at java.base/java.lang.Thread.run(Thread.java:834) The solution is to:

Rekey the MID Server Single Thread vs Multi-thread Shazzam Parameters mid.shazzam.threads and mid.shazzam.max_scanners_per_thread were added in Orlando. Parameter mid.shazzam.threads specifies the number of concurrent threads that Shazzam uses, while mid.shazzam.max_scanners_per_thread specifies the number of concurrent scanners processed by each Shazzam thread. Setting either of the parameter to 0 disables Shazzam multi-thread optimization. If any issues are found with the port scanning phase of discovery after upgrading from a pre Orlando release to Orlando+, try reverting to single threaded Shazzam.

Additional Information Discovery Phase Shazzam Shazzam probe causing node to run out of memory due to payload size Shazzam probes can be stuck in the ECC Queue output behind probes from later phases of discovery, due to running at the same Standard priority