Credentials & Permissions troubleshooting on Discovery, Service Mapping, Orchestration - Support and Troubleshooting

Credentials & Permissions troubleshooting on Discovery, Service Mapping, Orchestration Description This article provides info about specific credential types and troubleshooting steps for common credentials and permissions issues for Discovery, Orchestration, and Service Mapping.

For a walkthrough on troubleshooting credentials issues with Discovery, view the video below:

Credentials In order to discover a device or perform orchestration activities, add credentials to the credentials table. These credential records specify a username, a password, a kind of credential (Windows, SSH, ...), and MID Servers that are allowed to use this credential.

When the MID Server starts or when a credential is modified, the MID Server downloads and caches all available credentials.

Credentials for Discovery, Service Mapping, and Orchestration all point to the same credentials table.

Note: In case of credential failures, error message is changed from "Adding target to blacklist. No valid credential found for types..." to "Could not find any valid credentials to authenticate the target for types..." but there is no behavioral change. General Troubleshooting General troubleshooting will involve three main actions: Test the credential, ping the device, and Telnet into the port used for the credential test. See each section below for information about each action and steps to perform them correctly.

Test the credential Navigate to the credentials table. Select the credential. Click on Test credential . Populate the form fields for the credential test. Click OK . If the credential test fails, check that the credential has the correct username and password populated; a typographical error is very often the cause for the credential error. Once the credential fields are confirmed correct, if the credential error continues debugging can be turned on in the MID server for further investigation. The MID server logs should then be reviewed after the issue is reproduced with debug turned on. However, before turning debug confirm the MID server can communicate to the target as seen on the next steps.

Ping the device A ping confirms the device is available in the network.

Log into the MID server host, open up the command prompt and run: ping Please consult the target device administrator if ping is not successful.

Obs: Some environments may have ICMP requests disabled which could cause ping to fail.

Telnet into the port used for the credential test If a ping is successful, then the device is available in the network and reachable from the MID server host. However protocols used to communicate to the target hosts must connect to a specified port. Such ports may not be open on the target host or on the network path used to reach the device. A telnet test will confirm the whether the port is open or not.

telnet The following are some of the ports used out of box, however a system administrator could change these ports.

WMI: 135 SSH: 22 VCenter: 443 WinRM: 5985 WBEM: 5989 LDAP: 389

If the telnet test fails, an error message would be displayed, please consult the target device administrator and the network administrator to confirm that:

The firewall on the device allows traffic on the port tested by telnet There are no network firewalls blocking traffic from the MID server to the target host on the port tested Note: Telnet is an application that operates using the TCP protocol. UDP connectivity can not be tested using Telnet.

Collecting MID Server Debug logs To turn on debugging on the MID server, follow these steps:

Navigate to the MID server list, MID Server > Servers . Select the MID server used for the failed discovery or orchestration activity. Select the Configuration Parameters related list and click New . Select Parameter Name = mid.log.level and set the value to debug . Once debugging is complete, set this value back to info . To collect the MID server logs, follow these steps:

On the MID server record click on Grab MID logs . Click on the input records displayed. Windows Credentials A simple Powershell WMI query directly from the MID Server to the remote machine can be used to test access and permissions.

Open a PowerShell command line on the host where the MID server is being used and run the following:

gwmi win32_operatingsystem -computer 192.168.200.14 -credential 'LOCALDOMAIN\mid' Substitute LOCALDOMAIN\mid by the credential to test, and 192.168.200.14 with the target IP address. The expected result would be similar to:

SystemDirectory : C:\Windows\system32 Organization : BuildNumber : 6001 RegisteredUser : Windows User SerialNumber : 12345-OEM-1234567-12345 Version : 6.0.6001 If the WMI command above fails, either the credential is incorrect or lacks permission. We advise your windows admin team to further investigate the issue. If basic WMI queries from the MID server to the target hosts fails, then discovery and orchestration activities would not be successful.

Further Documentation on ServiceNow Windows Credentials Below is a collection of in-depth troubleshooting articles for windows credential errors.

KB0549834 - MID Server: troubleshooting WMI/Powershell issues - Credentials KB0549830 - Windows Discovery - Troubleshooting WMI/Powershell issues on the remote machine KB0549828 - WMI, PowerShell, and Windows Firewalls Below is a collection of product documentation articles containing detailed info about permission requirements for windows credentials.

Permission requirements for Windows credentials Discovery Windows probes and permissions Additional Discovery probe permissions SSH Credential Confirm Authentication To troubleshoot SSH credential errors, use any SSH client and connect to the target IP address from the MID server being used to ensure the account can successfully login to the target host, putty, for example. Make sure to use the same username/password combination as set in the credentials table. Access to the target system is necessary for any discovery or orchestration activity to work.

Confirm Authorization/Permission A user may be able to login to a target system however may not have permission to run the command attempted by either discovery or orchestration. See the following links and confirm whether the user meets the requirements.

Access Requirements for Non-Root Credentials UNIX and Linux commands requiring root privileges for Discovery and Orchestration To check what commands a user can run on a Unix based device, log into the target host with the user and run the following command:

sudo -l The output shows what privileged commands a user can run using sudo.

Check what implementation of SSH is being used mid.ssh.use_snc enables the ServiceNow SSH client (SNCSSH) on individual MID Servers. SNCSSH is a ServiceNow implementation of an SSH client and is active by default for all MID Servers on new instances, via a mid.ssh.use_snc. Enabling the ServiceNow SSH client disables the legacy J2SSH client.

Important: Mixing SSH client types for MID Servers connected to the same instance is not a good practice. Out of box true for new instances with Eureka or later and false for any prior existing instances. Private Key Credentials Sometimes sudo commands do not work with private key credentials, because there is no password to supply to the sudo command. A solution is to add the NOPASSWD option to the sudo configuration or provide a password to the credential. For example, you might enter: disco ALL=(root) NOPASSWD:/usr/sbin/dmidecode,/usr/sbin/lsof,/sbin/ifconfig. In the example above, the user disco can run dmidecode via sudo without providing a password. If a command fails when using a private key credential, confirm the user can run the command without providing a password.

Finally, if the account can successfully login to the target and execute the commands used by the probe being run or orchestration, however fails when using the MID server, follow the Collecting MID server logs section for further troubleshooting. Add parameter mid.ssh.debug = true for more details. SNMP Credential SNMP uses UDP, which does not create a virtual connection to the target host as TCP, and no reply may be given if a credential is not correct or authorized depending on the version used. A timeout can be seen in the discovery log when an invalid credential is used.

A couple of reasons for a timeout are:

The credential is not valid The network or target server is too busy and does not return the OIDs within the timeout Run an SNMP query to the target Using an SNMP tool, from the MID server, query OID 1.3.6.1.2.1.1.1. This OID is the sysDescr and will return a description of the device.

The following example uses SnmpWalk.exe, however has publi is an incorrect community string, the correct public string for this example should be public .

C:\SNMPWalk>.\SnmpWalk.exe -r:10.127.212.181 -c:"publi" -os:.1.3.6.1.2.1.1 -op:.1.3.6.1.2.1.1.1.0

%Failed to get value of SNMP variable. Timedout. As seen above there is no credential failure error. Instead of an error the query eventually times out.

In the following example the public string was corrected, public .

C:\SNMPWalk>.\SnmpWalk.exe -r:10.127.212.181 -c:"public" -os:.1.3.6.1.2.1.1 -op:.1.3.6.1.2.1.1.1.0

OID=.1.3.6.1.2.1.1.1.0, Type=OctetString, Value=Linux Linux-Tomcat 3.10.0-327.el7.x86_64 31 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 As seen above, once the public string was corrected then the sysDescr was returned, only part of it is shown above.

SNMP Parameters The "timeout" for an SNMP request can be increased via the "timeout" probe parameter. The default value is 1500, 1.5s. Try increasing the timeout if a credential is known to be correct and still no OIDs are returned. For tabular data try the "use_getbulk" parameter to improve efficiency.

See more information for SNMP probe parameters on SNMP probes .

No result returned from probe In some cases a warning can be seen in the discovery log stating No result returned from probe . This is seen when an input returns empty for the OID queried by the probe. This happens because a device does not have any results to return for the specified OID, and if that is the case this is expected. If there is suspicion that discovery may be wrong, a query for the same oid against the target IP address can be run to check on the output.

Further Documentation regarding SNMP credentials KB0696727: MID Server SNMP Troubleshooting VMWare Credential Confirm Authentication Confirm the same account configured in the credentials table can log into the VCenter target:

Log into the MID server host Open up a browser and navigate to https:// /mob , replace the address with the IP address of the VCenter server Make sure to use the same exact username/password combination and the same format as seen in the credentials table record If the test above fails have your vmware team further troubleshoot or provide access to the credential. If the test above is successful, however discovery still fails, then follow the steps for Collecting MID server debug logs for further investigation.

Further Documentation VCenter Discovery Other Credentials UCS-HD