REST Message via MID Server - Unable to decrypt parameter: rest_password, causes 401 - Invalid username/password combo


Description

REST Message via MID Server can fail due to the MID Server being unable to decrypt the password being sent to use with the job. Deleting agent\keystore\agent_keystore.jks file and re-Validating the MID Server seems to resolve this, suggesting something has gone wrong with the encryption keys shared between instance and MID Server.

The REST Message will fail with "[ERROR CODE: 401 ] Method failed: (/2/query) with code: 401 - Invalid username/password combo"
The MID Server agent logs will show errors from the MIDWorker thread: "Unable to decrypt parameter: rest_password, using encrypted value SNC_ENC_VAL[...." and "Unable to decrypt password".

Steps to Reproduce

The steps to get the MID Server into the state where it cannot decrypt the password is currently unknown.

When a REST Message is sent once the MID Server is in this state, the REST Message will fail with ""[ERROR CODE: 401 ] Method failed: (/2/query) with code: 401 - Invalid username/password combo", and something along the lines of this will be seen in the instance app node:

2019-02-01 05:15:56 (307) worker.5 worker.5 txid=b2f4dae6dbdb OUTBOUND_HTTP: protocol= response_status=401 response_time=5969 request_length=0 response_length=85 app_scope=XXXX session_id=XXXX transaction_name="#7188751 /sys_rest_message_fn.do" user_name=dravvy.ramlochun@snc mid_server=XXXX source_table=sys_ui_action source_record=48c17ed207131000dada43c0d1021e83 system_id=XXXX method=GET log_level=All scheme=https hostname=XXXXpath=/2/query url=https://XXXX:1671/2/query?platform=windows&query=(select%20(name)%20(from%20device))&format=xml response_body="[ERROR CODE: 401 ] Method failed: (/2/query) with code: 401 - Invalid username/password combo" request_headers={} response_headers="{Transfer-Encoding=chunked, WWW-Authenticate=Basic realm="Credentials", Connection=Keep-Alive, Content-Type=text/html}" request_query=platform=windows&query=(select%20(name)%20(from%20device))&format=xml request_body= url=https://XXXX:1671/2/query?platform=windows&query=(select%20(name)%20(from%20device))&format=xml

In the MID Server agent logs, an exception like this will be seen, showing the true cause of the error, which is that the MID Server could not properly decrypt the password sent from the instance:

09/17/18 10:21:39 (311) Worker-Standard:MIDWorker WARNING *** WARNING *** Unable to decrypt parameter: rest_password, using encrypted value SNC_ENC_VAL[dOi5occXD6YcOcqaVYslxrVHQ3P/OlimQIW/E2TB4Vgy] instead : Index: 0, Size: 0
09/17/18 10:21:39 (311) Worker-Standard:MIDWorker SEVERE *** ERROR *** Unable to decrypt password
com.snc.automation_common.integration.exceptions.EncryptionException: Index: 0, Size: 0
at com.snc.commons.ParameterEncrypter.getEncrypter(ParameterEncrypter.java:310)
at com.snc.commons.ParameterEncrypter.decryptString(ParameterEncrypter.java:289)
at com.snc.commons.ParameterEncrypter.decrypt(ParameterEncrypter.java:273)
at com.snc.commons.ParameterEncrypter.decrypt(ParameterEncrypter.java:257)
at com.snc.commons.ParameterEncrypter.decrypt(ParameterEncrypter.java:187)
at com.snc.commons.eccprobe.RESTProbe.getPassword(RESTProbe.java:442)
at com.snc.commons.eccprobe.RESTProbe.getHttpRequestFromPayload(RESTProbe.java:343)
at com.snc.commons.eccprobe.RESTProbe.probe(RESTProbe.java:164)
at com.service_now.mid.probe.ECCProbeProxy.probe(ECCProbeProxy.java:45)
at com.service_now.mid.probe.AProbe.process(AProbe.java:84)
at com.snc.commons.eccprobe.AECCProbe.process(AECCProbe.java:82)
at com.service_now.mid.queue_worker.MIDWorker.process(MIDWorker.java:27)
at com.service_now.mid.queue_worker.AWorker.runWorker(AWorker.java:125)
at com.service_now.mid.queue_worker.AWorkerThread.run(AWorkerThread.java:20)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at com.glide.util.AutomationEncryptionKeyProvider.getCurrentKey(AutomationEncryptionKeyProvider.java:26)
at com.glide.util.AutomationEncrypter.<init>(AutomationEncrypter.java:48)
at com.snc.commons.ParameterEncrypter.getEncrypter(ParameterEncrypter.java:308)
... 16 more

Workaround

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.

Where this issue has been seen, re-validation of the MID Server seems to resolve the issue:

  1. From the host server which has the MID Server installed:
    • Stop the MID Server service
    • Delete the agent\keystore\agent_keystore.jks file, from the installation folder.
    • Start the MID Server service
  2. From the Instance
    1. Open the MID Server record
    2. (If necessary, wait for the MID Server to be UP and then reload the form, so all the Related Links are shown)
    3. Click 'Validate' in the Related Links

The MID Server will now restart, and you should now be able to run the REST Message successfully. 


Related Problem: PRB1361865