How to troubleshoot MID Server WMI and PowerShell credential issues for Windows Discovery - Support and Troubleshooting

How to troubleshoot MID Server WMI and PowerShell credential issues for Windows Discovery Issue .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Troubleshoot credential failures in Discovery Logs that occur after running a Discovery Schedule for Windows machines. In some cases, the Discovery Logs indicate a credential failure but the actual cause is external to the MID Server. This article explains how Windows probes use credentials and provides troubleshooting steps for common WMI and PowerShell errors.

Facts .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Credentials To discover target Windows machines, add Windows credentials to the Credentials table. Credential records specify a user name, a password, a credential type (Windows, SSH, SNMP), and the MID Servers allowed to use the credential.

When the MID Server starts or when a credential is modified, the MID Server downloads and caches all available credentials.

MID Server configuration parameters The three main configuration parameters covered in this article are:

mid.use_powershell —When set to true, the MID Server retrieves WMI and Windows registry values using the PowerShell probe. mid.powershell.use_credentials —Enables the PowerShell probe to use credentials defined in the Credentials table. mid.powershell.local_mid_service_credential_fallback —Enables the PowerShell probe to use the local MID account as a last resort. Beginning with the Fuji release, these parameters are true by default.

Running Windows probes Many Windows probes retrieve WMI and Windows registry values from the target machine. There are two methods, depending on the value of the mid.use_powershell configuration parameter.

Running the WMIRunner probe This probe cannot use the credentials defined in Discovery > Credentials . The MID Server connects to the WMI providers using the account that the MID Server service runs with, referred to as the local MID account. This account could be a domain user account. The local MID account must have access to the remote machines for the WMIRunner probe to work.

Running the PowerShell probe This probe can use the credentials defined in Discovery > Credentials . The MID Server uses the PowerShell API to retrieve WMI values. If the MID Server is configured to run PowerShell probes using credentials (see the MID Server configuration parameters section), it attempts to connect to the target machine using the first defined credential. If that credential fails, the next credential is attempted.

If all credentials fail and the MID Server is configured to use the local MID account as a last resort, the MID Server attempts to connect with the account the MID Server service runs with.

Simulating a simple probe Go to System Maintenance > Scripts - Background . Run the following script, replacing the mid_server and target_host variables with the name of your MID Server and the IP address of the target Windows machine: var mid_server = 'MID_SERVER_1' ; var target_host = '192.168.200.14' ; var debug = true ; sendProbeWmiRunner(mid_server, target_host); function sendProbeWmiRunner(mid, host) { var ecc = new GlideRecord( 'ecc_queue' ); ecc.initialize(); ecc.agent = 'mid.server.' + mid; ecc.topic = 'WMIRunner' ; ecc.name = 'TestWMI' ; ecc.source = host; ecc.queue = 'output' ; ecc.payload = ' + debug + '"/> ' ; ecc.insert(); } The script sends a WMIRunner probe to the MID Server that retrieves the serial number from the target machine. If mid.use_powershell is true, the MID Server switches the WMIRunner probe internally to PowerShell.

Release .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All supported releases

Resolution .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Troubleshooting Windows probes If errors appear in the Discovery Logs after running WMI or PowerShell probes, check the MID Server configuration parameter mid.use_powershell:

If mid.use_powershell is false, see the Troubleshooting WMIRunner probes section. If mid.use_powershell is true, see the Troubleshooting PowerShell probes section. Troubleshooting WMIRunner probes - mid.use_powershell:false Send the probe (see the Simulating a simple probe section). Go to Discovery > ECC Queue and sort the list by Created from newest to oldest. Locate the output message with topic "WMIRunner" and name "TestWMI." Wait a few seconds for the MID Server to process the probe. Refresh the list until the response ECC message appears. Open the message and review the contents of Payload. Select the XML button to view the formatted payload. Review the message within the result tag. When the probe succeeds, the result is similar to the following:

VMware-56 4d 41 9f 34 6e 0c 3d-1b be f1 3c f7 2c ad 7b

When the credential is incorrect, the result is similar to the following:

Connection failed to WMI service. Error: Permission denied Thu May 28 16:24:21 2015 DEBUG: Testing WMI connection to 192.168.200.14 Thu May 28 16:25:03 2015 DEBUG: Appending element: error Thu May 28 16:25:03 2015 DEBUG: getValueText() Thu May 28 16:25:03 2015 DEBUG: valueType: string Thu May 28 16:25:03 2015 DEBUG: Appended: error; Connection failed to WMI service. Error: Permission denied Thu May 28 16:25:03 2015 DEBUG: WMI not running This error indicates that the account the MID Server service runs with is not a valid credential for the WMI API to connect to the target machine.

Solution There are two options:

Set mid.use_powershell to true. This is the preferred option. Set the account the MID Server service runs with (Windows service) to have the correct permissions to access the remote machine. Troubleshooting PowerShell probes - mid.use_powershell:true Send the probe (see the Simulating a simple probe section). Go to Discovery > ECC Queue and sort the list by Created from newest to oldest. Locate the output message with topic "WMIRunner" and name "TestWMI." Wait a few seconds for the MID Server to process the probe. Refresh the list until the response ECC message appears. Open the message and review the contents of Payload. Select the XML button to view the formatted payload. Review the message within the result tag.

When the probe succeeds, the result is similar to the following:

VMware-56 4d 41 9f 34 6e 0c 3d-1b be f1 3c f7 2c ad 7b The following sections describe the most common errors.

The RPC server is unavailable When mid.powershell.local_mid_service_credential_fallback is true, the error appears as:

Authentication failure with the local MID server service credential. Failed to access target system. Please check credentials and firewall settings on the target system to ensure accessibility: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) Stack Trace: at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Management.ManagementScope.InitializeGuts(Object o) at System.Management.ManagementScope.Initialize() at System.Management.ManagementObjectSearcher.Initialize() at System.Management.ManagementObjectSearcher.Get() at Microsoft.PowerShell.Commands.GetWmiObjectCommand.BeginProcessing()

When mid.powershell.local_mid_service_credential_fallback is false, the error appears as:

Failure(s) with available Windows credentials from the instance. Credentials tried: LOCALDOMAIN\mid,autolab1\autouser The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) Stack Trace: at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Management.ManagementScope.InitializeGuts(Object o) at System.Management.ManagementScope.Initialize() at System.Management.ManagementObjectSearcher.Initialize() at System.Management.ManagementObjectSearcher.Get() at Microsoft.PowerShell.Commands.GetWmiObjectCommand.BeginProcessing() This error can be misleading. In most cases, it is not an authentication failure caused by an incorrect credential.

In the MID Server logs with debug enabled:

The MID Server first tries the credentials from the Credentials table. These fail with exit code 2 (when mid.powershell.use_credentials is true). Then it tries the account the MID Server service runs with. This fails with exit code 3 (when mid.powershell.local_mid_service_credential_fallback is true). In all cases, the error indicates that the RPC server is unavailable (Exception from HRESULT: 0x800706BA). The MID Server returns the error details from the last credential attempted. This indicates that the MID Server cannot access the remote machine using RPC, typically caused by a Windows firewall on the remote machine blocking RPC requests.

Troubleshooting Run a PowerShell WMI query directly from the MID Server to the remote machine. Open a PowerShell command line and run the following, replacing LOCALDOMAIN\mid with the credential to test:

gwmi win32_operatingsystem -computer 192.168.200.14 -credential 'LOCALDOMAIN\mid' The expected result is similar to:

SystemDirectory : C:\Windows\system32 Organization : BuildNumber : 6001 RegisteredUser : Windows User SerialNumber : 12345-OEM-1234567-12345 Version : 6.0.6001 In this case, the following error is returned:

Get-WmiObject : The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) At line:1 char:5 + gwmi Solution Verify IP connectivity to the remote machine using ping. If the ping fails, there is a routing problem. Contact your network administrator. If the ping succeeds, the issue is related to the remote machine or a filtering device between the MID Server and the remote machine. Fix the firewall configuration on the remote machine. See Opening ports in Windows Firewall for remote server access for information about troubleshooting RPC and DCOM related issues. Access is denied When mid.powershell.local_mid_service_credential_fallback is true, the error appears as:

Authentication failure with the local MID server service credential. Failed to access target system. Please check credentials and firewall settings on the target system to ensure accessibility: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) Stack Trace: at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Management.ManagementScope.InitializeGuts(Object o) at System.Management.ManagementScope.Initialize() at System.Management.ManagementObjectSearcher.Initialize() at System.Management.ManagementObjectSearcher.Get() at Microsoft.PowerShell.Commands.GetWmiObjectCommand.BeginProcessing() at System.Management.Automation.Cmdlet.DoBeginProcessing() at System.Management.Automation.CommandProcessorBase.DoBegin() When mid.powershell.local_mid_service_credential_fallback is false, the error appears as:

Authentication failure(s) with available Windows credentials from the instance. Credentials tried: autolab1\autouserBAD,LOCALDOMAIN\midBAD If the local MID service credential is not enabled, the E_ACCESSDENIED error message does not appear. The authentication failure message is reliable. In most cases, this error is caused by an incorrect credential.

In the MID Server logs with debug enabled:

The MID Server first tries the credentials from the Credentials table. These fail with exit code 1 (when mid.powershell.use_credentials is true). Then it tries the account the MID Server service runs with. This fails with exit code 3 (when mid.powershell.local_mid_service_credential_fallback is true). The MID Server returns the error details from the last credential attempted. The MID Server logs return a different error for remote and local credentials, which can be misleading. Complete the following troubleshooting steps to confirm this is an incorrect credential issue.

Troubleshooting Run a PowerShell WMI query directly from the MID Server to the remote machine. Open a PowerShell command line and run the following, replacing LOCALDOMAIN\mid with the credential to test:

gwmi win32_operatingsystem -computer 192.168.200.14 -credential 'LOCALDOMAIN\mid' The expected result is similar to:

SystemDirectory : C:\Windows\system32 Organization : BuildNumber : 6001 RegisteredUser : Windows User SerialNumber : 12345-OEM-1234567-12345 Version : 6.0.6001 In this case, the following error is returned:

Get-WmiObject : Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) At line:1 char:5 + gwmi This confirms that the credential (user name, password, or both) is incorrect.

Solution Find a credential that works and update it in ServiceNow.

Call was canceled by the message filter When mid.powershell.local_mid_service_credential_fallback is true, the error appears as:

Authentication failure with the local MID server service credential. Call was canceled by the message filter. (Exception from HRESULT: 0x80010002 (RPC_E_CALL_CANCELED)) Stack Trace: at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Management.ManagementScope.InitializeGuts(Object o) at System.Management.ManagementScope.Initialize() at System.Management.ManagementObjectSearcher.Initialize() at System.Management.ManagementObjectSearcher.Get() at Microsoft.PowerShell.Commands.GetWmiObjectCommand.BeginProcessing()

When mid.powershell.local_mid_service_credential_fallback is false, the error appears as:

Failure(s) with available Windows credentials from the instance. Credentials tried: AUTOLAB1\autouser,localdomain\mid Call was canceled by the message filter. (Exception from HRESULT: 0x80010002 (RPC_E_CALL_CANCELED)) Stack Trace: at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Management.ManagementScope.InitializeGuts(Object o) at System.Management.ManagementScope.Initialize() at System.Management.ManagementObjectSearcher.Initialize() at System.Management.ManagementObjectSearcher.Get() at Microsoft.PowerShell.Commands.GetWmiObjectCommand.BeginProcessing() Troubleshooting The method for troubleshooting this error is similar to the methods used for the previous errors. Run the PowerShell test command and determine if the error can be reproduced. If it can, the issue is between the MID Server and the remote machine.

This error indicates that the RPC connection is being canceled by the remote machine. Try running the same command against other Windows machines to determine if it works. If it does, this confirms the problem is isolated to one particular Windows machine. The cause on the remote machine could be the Windows firewall filtering access to RPC/DCOM.

Solution The following articles can help troubleshoot and fix these problems:

Opening ports in Windows Firewall for remote server access How to troubleshoot WMI and PowerShell issues on a remote machine for Windows Discovery Server execution failed When mid.powershell.local_mid_service_credential_fallback is true, the error appears as:

Authentication failure with the local MID server service credential. Failed to access target system. Please check credentials and firewall settings on the target system to ensure accessibility: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) Stack Trace: at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Management.ManagementScope.InitializeGuts(Object o) at System.Management.ManagementScope.Initialize() at System.Management.ManagementObjectSearcher.Initialize() at System.Management.ManagementObjectSearcher.Get() at Microsoft.PowerShell.Commands.GetWmiObjectCommand.BeginProcessing() at System.Management.Automation.Cmdlet.DoBeginProcessing() at System.Management.Automation.CommandProcessorBase.DoBegin()

When mid.powershell.local_mid_service_credential_fallback is false, the error appears as:

Failure(s) with available Windows credentials from the instance. Credentials tried: autolab1\autouser,LOCALDOMAIN\mid Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE)) Stack Trace: at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) at System.Management.ManagementScope.InitializeGuts(Object o) at System.Management.ManagementScope.Initialize() at System.Management.ManagementObjectSearcher.Initialize() at System.Management.ManagementObjectSearcher.Get() at Microsoft.PowerShell.Commands.GetWmiObjectCommand.BeginProcessing() If an incorrect credential is used, the E_ACCESSDENIED error message appears. If a valid credential is used, the CO_E_SERVER_EXEC_FAILURE error message appears. This occurs when the Windows Management Instrumentation service is paused or stopped on the remote machine.

Troubleshooting Open a PowerShell command line on the MID Server and run the following, replacing LOCALDOMAIN\mid with the credential to test:

gwmi win32_operatingsystem -computer 192.168.200.14 -credential 'LOCALDOMAIN\mid' The expected result is similar to:

SystemDirectory : C:\Windows\system32 Organization : BuildNumber : 6001 RegisteredUser : Windows User SerialNumber : 12345-OEM-1234567-12345 Version : 6.0.6001 In this case, the following error is returned:

Get-WmiObject : Server execution failed (Exception from HRESULT: 0x80080005 (CO_E_SERVER_EXEC_FAILURE)) At line:1 char:5 + gwmi This issue is typically caused by the Windows Management Instrumentation service being stopped on the remote machine.

Solution Start the Windows Management Instrumentation on the remote machine.