Debugging Credentials sync errors in the MID Server, and the GetMIDInfo/getCredentials SOAP requests they rely onSummary Many different features/apps use the MID Server to do integrations, and use the Credentials from the table in the instance to authenticate with the endpoints. There are many causes of why the MID Server is not able to get those credentials from the instance before it can use them with those probes. This KB aims to give general tips on how to debug this kind of issue. Note: Checked for corrupt records in the discovery_credentials table, and the MID Server is validated with a good keychain certificate, before going this far, as in 2021 those are much more likely to be the causes. Example Errors The "Failed to retrieve credentials and affinities" error seen in the agent log, for example below when running a PowerShell step from an IntegrationHub AD Spoke for a Flow Designer workflow. The orange errors are just a side effect of the earlier yellow errors when failing to fetch the credentials from the instance. 12/07/21 09:28:03 (580) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 Worker starting: IPaaSActionProbe source: 2188411d601c4598654314fee866757d12/07/21 09:28:04 (549) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: Received 2188411d601c4598654314fee866757d but no listeners found, proceeding with default12/07/21 09:28:04 (642) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 Found local IP addresses: 127.0.0.1,10.110.117.23512/07/21 09:28:04 (924) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: Flushing session cache due to credential load.12/07/21 09:28:04 (939) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 Creating cache...12/07/21 09:30:59 (977) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 WARNING *** WARNING *** Socket timeout12/07/21 09:30:59 (977) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 SEVERE *** ERROR *** SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><agent xsi:type="xsd:string">be0d0a89f59c059818d469c47e912e1f</agent> <purpose xsi:type="xsd:string">GetCredentials</purpose></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>12/07/21 09:30:59 (993) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 SEVERE *** ERROR *** SOAP Response: Status code=0, Response body=null12/07/21 09:30:59 (993) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 SEVERE *** ERROR *** GetCredentials: non-retryable html response - Socket timeout12/07/21 09:30:59 (993) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 SEVERE *** ERROR *** Failed to retrieve credentials and affinities.12/07/21 09:30:59 (993) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: No valid credential found, running script without credential12/07/21 09:30:59 (993) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: Executing command: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -executionpolicy unrestricted -noninteractive -nologo -noprofile -command "& {& 'scripts\PowerShell\PSScript.ps1' -computer '10.110.117.112' -script 'C:\ServiceNow\MID Server Rct\agent3\scripts\PowerShell\ADSpoke\ActionLookupGroup.ps1' -useCred $true -ismid $false -isDiscovery $true -debug $true -logInfo $false -skipTest $false -executeRemote $false -processTimeout 600 -copyScriptToTarget $false; exit $LASTEXITCODE}"12/07/21 09:30:59 (993) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: With credential named : MID service account12/07/21 09:31:00 (039) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: Thread name is Powershell is executing...12/07/21 09:31:01 (383) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: The exit value from waitFor() is 312/07/21 09:31:03 (462) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 DEBUG: Execution status: failed12/07/21 09:31:03 (462) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 SEVERE *** ERROR *** Authentication failure with the local MID server service credential12/07/21 09:31:03 (462) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 SEVERE *** ERROR *** Failed while executing ActionLookupGroup.ps1 (Access denied)12/07/21 09:31:03 (462) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 SEVERE *** ERROR *** Authentication failure with the local MID server service credential12/07/21 09:31:03 (462) Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9 2021-12-07 09:30:59 Tried credential: MID service account, status=failed... Or an error like this running a Discovery Shazzam probe 2021-12-14 08:57:29 (067) ConcurrentPortScannerEngine-0:7b300be7879801d02c8864e83cbb3538 SEVERE *** ERROR *** An error occurred while decrypting credentials from instancecom.snc.automation_common.integration.exceptions.AutomationIOException: Unable to retrieve data from instance. This MID may not be validated.at com.glide.util.MIDServerInfoPayloadDecrypter.decryptPayload(MIDServerInfoPayloadDecrypter.java:25)at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.loadCredentials(StandardCredentialsProvider.java:317)at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.load(StandardCredentialsProvider.java:287)at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.loadIfNecessary(StandardCredentialsProvider.java:299) And in that case was probably caused by 429 rejections like below running this call to GetMIDInfo GetIsAgentValid during a credential_reload System Command, which also would mean the credentials are not loaded. 2021-12-12 23:43:31 (253) Worker-Interactive:SystemCommand-6bd7bf83871c81502c8864e83cbb351c DEBUG: Event: RGRPerfMetricEvent2021-12-12 23:43:31 (424) Worker-Interactive:SystemCommand-6bd7bf83871c81502c8864e83cbb351c SEVERE *** ERROR *** SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><agent xsi:type="xsd:string">ad85a9eddb061810ccbe44c3059619f1</agent><purpose xsi:type="xsd:string">GetIsAgentValid</purpose></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>2021-12-12 23:43:31 (424) Worker-Interactive:SystemCommand-6bd7bf83871c81502c8864e83cbb351c SEVERE *** ERROR *** SOAP Response: Status code=429, Response body=<!DOCTYPE html><html><head><title>Error Page</title></head><body><h2 align="center" style="margin:10%;">Sorry, an error occurred or this page isn't available.</h2></body></html>2021-12-12 23:43:31 (424) Worker-Interactive:SystemCommand-6bd7bf83871c81502c8864e83cbb351c WARNING *** WARNING *** GetIsAgentValid: request failed - null waiting 10 seconds to retry2021-12-12 23:43:42 (018) Worker-Interactive:SystemCommand-6bd7bf83871c81502c8864e83cbb351c DEBUG: Flushing session cache due to credential load.2021-12-12 23:43:43 (237) Worker-Interactive:SystemCommand-6bd7bf83871c81502c8864e83cbb351c SEVERE *** ERROR *** An error occurred while decrypting credentials from instance This example is due to bad API Key credential record in the instance 2023-09-03T07:51:46.680-0400 INFO (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [AWorker:128] Worker starting: SystemCommand source: credentials_reload2023-09-03T07:51:46.680-0400 INFO (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [Instance:928] Running system command: credentials_reload2023-09-03T07:51:49.111-0400 WARN (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [XMLUtil:523] org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 170660; Character reference "&#16" is an invalid XML character.2023-09-03T07:51:49.111-0400 WARN (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [StandardCredentialsProvider:402] Problem reading credentials: null2023-09-03T07:51:49.111-0400 WARN (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [StandardCredentialsProvider:385] No credentials from instance were loaded. MID server may not have access to credentials table2023-09-03T07:51:49.111-0400 INFO (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [MIDCredentialsConfigProvider$MyEventListener:83] com.service_now.mid.creds.provider.standard.StandardCredentialsProvider reloaded2023-09-03T07:51:49.111-0400 INFO (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [ECCSender:206] Enqueuing: E:\\agent\work\monitors\ECCSender\output_0\ecc_queue.95b008394745b194d7327942846d4329.xml2023-09-03T07:51:49.127-0400 INFO (Worker-Interactive:SystemCommand-95b008394745b194d7327942846d4329) [AWorker:136] Worker completed: SystemCommand source: credentials_reload time: 0:00:02.431 Instance app node logs to go with the GetMIDInfo request, showing the problems the instance had decrypting the credential before sending it in the request response. 2023-09-03 03:56:32 (491) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 WARNING *** WARNING *** REST Msg Outbound - OAuthTokenRetriever : No token in OAuth2Profile: [XTM,b1cd40c673521010395108b24ff6a7b3]2023-09-03 03:56:32 (801) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 WARNING *** WARNING *** string may not be encrypted: Error occurred while trying to: SYMMETRIC_DECRYPTION2023-09-03 03:56:32 (801) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 WARNING *** WARNING *** Value present in the column: api_key from table: api_key_credentials may not be encrypted2023-09-03 03:56:32 (801) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 Reloading script engine cache2023-09-03 03:56:32 (805) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 SEVERE *** ERROR *** Couldn't find Crypto Module2023-09-03 03:56:32 (813) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 WARNING *** WARNING *** Module key for the missing Crypto Module is: d2532d131b858594235015ff034bcb2c2023-09-03 03:56:32 (814) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 WARNING *** WARNING *** string may not be encrypted: Input length must be multiple of 8 when decrypting with padded cipher2023-09-03 03:56:32 (814) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 WARNING *** WARNING *** Value present in the column: password from table: discovery_credentials may not be encrypted2023-09-03 03:56:33 (033) SOAPProcessorThread8814b7691b89fd1460ae54a3604bcb60 025135AD1B853D1460AE54A3604BCBF8 txid=8c147f691b89 WARNING *** WARNING *** REST Msg Outbound - OAuthTokenRetriever : No token in OAuth2Profile: [RWS,227c722573521010395108b24ff6a78b] Showing the XML of the bad api_key_credentials record confirms there were some weird characters in the API Key field to cause this, which was not in the normal encrypted field format.<api_key>���A��0f���7r��{7!��</api_key>Instructions Table of Contents Turn on MID Server debug, and reproduceUnderstand what code is handling the request in the instanceFind the TransactionTurn on SOAP debug, and check the instance app node logsRun the same code from a background scriptSession Debug can be turned onMonitor stats.do / Threads.doRoot cause in this example: Turn on MID Server debug, and reproduce The above example log was from a test run when the MID Server had parameter mid.log.level=trace set, which is why you can see the additional DEBUG and TRACE log lines. If you can reproduce an issue when a probe runs in a mid server, then reproduce it will full logging as usually the clues you will need are there in the additional logging. "Worker-Expedited:IPaaSActionProbe-e588811df51c459818d469c47e912ea9" means IPaaSActionProbe is the IntegrationHub probe code that's running in this thread, and the sys_id in the thread name is the ecc_queue output record. This particular probe does not tend to put much debug in the ecc_queue input record, or flow context, and so MID Server agent logs are needed. This also applies to other features like Discovery, where only a high level error for the feature is given, and not the true underlying errors that caused it. The first error is usually the important one. We need to understand what this means: WARNING *** Socket timeout SEVERE *** ERROR *** SOAP Request: ... xmlns:tns="http://www.service-now.com/GetMIDInfo" ... <purpose xsi:type="xsd:string">GetCredentials</purpose> ... The warning says a socket timeout occurred, which is a network error to do with a connection. It is a fair assumption that this socket error is the cause of the ERROR. That error message tells us a SOAP request, to the instance (www.service-now.com means this is a servicenow API), to a URL "/GetMIDInfo", which is a Scripted SOAP Web Service. Understand what code is handling the request in the instance With the "GetMIDInfo" Scripted SOAP Web Service record open in the instance, we can see the description is "Web Service to access MID Server credentials, affinities, and other relevant information", which fits with the later errors. Inspect the script and we see "GetCredentials" is one of the functions this web service can do. The timestamps on the log entry before the socket timeout and error are also useful 09:28:04 (939) "Creating cache", which we can probably assume is the credential cache, then errors at 12/07/21 09:30:59 (977) = nearly 3 minutes, with nothing apparently happening in the MID Server, so we now need to think about the instance end of this transaction. Find the Transaction We can now look in the instance Transaction log to find this and other similar transactions for this web service. This will tell us metrics about the transaction, such as the wait time, runtime, database queries, plus also the instance app node and session ID as that will be needed to find the transaction in the app node logs later. syslog_transaction is a big table, so it's best to start with just the filter: https://<instance>.service-now.com/syslog_transaction_list.do?sysparm_filter_only=true Then add conditions such as Created on today, type=SOAP, URL starts with /GetMIDInfo https://<instance>.service-now.com/syslog_transaction_list.do?sysparm_query=sys_created_onONToday%40javascript%3Ags.beginningOfToday()%40javascript%3Ags.endOfToday()%5Etype%3Dsoap%5EurlSTARTSWITH%2FGetMIDInfo&sysparm_view= The Created timestamp is the End of the transaction. If you can match your MID agent log error to a transaction record, you will know which app node log to look in, and which session to search for. The transaction record also gives the specific transaction number, e.g. 134493, which you can match to the tx_id in the app node localhost log. Turn on SOAP debug, and check the instance app node logs To capture the incoming SOAP envelope XML in the system log, which should confirm the mid server name and other inputs to the web service, add the property glide.processor.debug.SOAPProcessor with a value of true.When enabled, this property also adds the incoming SOAP envelope in the Message field of the system log (System Logs > All). You may find there is an exception? e.g. 2021-06-30 15:39:59 (677) API_INT-thread-5 SYSTEM txid=592d84c61b4d WARNING *** WARNING *** Cookie token not in database: SCv3:PN+WsAqa+n6cVGtCb3besmBqc5BnA52k:yQIdHtktAzjEa0mQ46RhmTL947p7cVSMalt5lwzHdm4=2021-06-30 15:39:59 (677) API_INT-thread-5 SYSTEM txid=592d84c61b4d WARNING *** WARNING *** aborted activity cookie update: U0N2MzpQTitXc0FxYStuNmNWR3RDYjNiZXNtQnFjNUJuQTUyazp5UUlkSHRrdEF6akVhMG1RNDZSaG1UTDk0N3A3Y1ZTTWFsdDVsd3pIZG00PQ==2021-06-30 15:39:59 (678) API_INT-thread-5 E3F1D1BD1B49B014E1E09979B04BCBDF txid=592d84c61b4d *** Start #4044724 /GetMIDInfo.do, user: datacentertest_bfi0_disco_midserver82021-06-30 15:39:59 (756) SOAPProcessorThread992d08ca1b4df414e1e09979b04bcbe3 E3F1D1BD1B49B014E1E09979B04BCBDF txid=9d2dc8ca1b4d WARNING *** WARNING *** string may not be encrypted : Input length must be multiple of 8 when decrypting with padded cipher2021-06-30 15:39:59 (770) SOAPProcessorThread992d08ca1b4df414e1e09979b04bcbe3 E3F1D1BD1B49B014E1E09979B04BCBDF txid=9d2dc8ca1b4d WARNING *** WARNING *** GlideRecord newGlideRecord() called with invalid table name: u_temp_test_unknown_credentials2021-06-30 15:39:59 (770) SOAPProcessorThread992d08ca1b4df414e1e09979b04bcbe3 E3F1D1BD1B49B014E1E09979B04BCBDF txid=9d2dc8ca1b4d SEVERE *** ERROR *** loadRow failurejava.lang.NullPointerException at com.glide.script.GlideRecord.loadRow(GlideRecord.java:3557) at com.glide.script.GlideRecord.initIfNecessary(GlideRecord.java:1039) at com.glide.script.GlideRecord.getElement(GlideRecord.java:7338) at com.glide.script.GlideRecord.getValue(GlideRecord.java:7046) at com.glide.script.MIDServerInfoAccessor.skipCredential(MIDServerInfoAccessor.java:715) at com.glide.script.MIDServerInfoAccessor.loadCredentials(MIDServerInfoAccessor.java:553)... Hopefully there is an obvious error in the app node logs to explain it. If not, there are system properties such as glide.db.trace that will add logs for every database query. Disable these debugging feature as soon as you are finished so that the log is not overwhelmed with excessive and unnecessary debugging information. Run the same code from a background script If you do see that there is a problem running the web service in the instance, then you could run the same code in a background script. Scripts - Background (/sys.scripts.do).In my example here, I have cut down the script from the web service to a minimum, and "doc" is basically what gets sent back to the mid server in the response: var MIDServerInfoAccessor = new GlideMIDServerInfoAccessor();GlideSession.get().putProperty("MidCallbackOnCredentialsReload", true);var doc = MIDServerInfoAccessor.getCredentialXML("317050c51bd80510cdec62c7bd4bcb7f");GlideSession.get().putProperty("MidCallbackOnCredentialsReload", false); 317050c51bd80510cdec62c7bd4bcb7f is the sys_id of my MID Server's ecc_agent record. You would need to replace that for your mid server record. Output may look something like this, even if it is 'working' as it was in my example, but if there are problems it may take a long time to complete, or provide other errors. You might need to impersonate the MID Server user in order to reproduce the issue, in which case, wrap the script in: var userGR = new GlideRecord('sys_user');userGR.get('user_name', "david.piper"); // put user_id herevar originalUser = gs.getSession().impersonate(userGR.sys_id);//// Code goes here.//gs.getSession().impersonate(originalUser); Session Debug can be turned on Session Debug can be turned on while you run the background script to see all the query business rules, and database queries etc. that run while this script runs. That may help you identify exactly where a performance issue lies. Monitor stats.do / Threads.do Long running jobs can be monitored from the /stats.do /threads.do pages. In /stats.do, a MID server request will be in the API_INT semaphore section as /GetMIDInfo.do, or if running a background script will be /sys.scripts.do in the default semaphore sets. Clicking the thread link regularly, to open in new browser tabs, can give a good picture of what code is running over time. e.g. here is the thread clearly spending time in the Pseudorandom Number generator functions. main,Default-thread-1203,5,attrs=(session_id=273374F4BD30095C18D40EE589639862) java.io.FileInputStream.readBytes(Native Method) java.io.FileInputStream.read(FileInputStream.java:255) sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424) sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:441) sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:331) sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:226) java.security.SecureRandom.generateSeed(SecureRandom.java:533) org.bouncycastle.crypto.util.BasicEntropySourceProvider$1.getEntropy(Unknown Source) org.bouncycastle.crypto.fips.ContinuousTestingEntropySource.getEntropy(Unknown Source) org.bouncycastle.crypto.fips.HashSP800DRBG.getEntropy(Unknown Source) org.bouncycastle.crypto.fips.HashSP800DRBG.init(Unknown Source) org.bouncycastle.crypto.fips.HashSP800DRBG.(Unknown Source) org.bouncycastle.crypto.fips.FipsDRBG$HashDRBGProvider.get(Unknown Source) org.bouncycastle.crypto.fips.DRBGPseudoRandom.lazyInitDRBG(Unknown Source) org.bouncycastle.crypto.fips.DRBGPseudoRandom.generate(Unknown Source) org.bouncycastle.crypto.fips.FipsSecureRandom$RandomSpi.engineNextBytes(Unknown Source) java.security.SecureRandom.nextBytes(SecureRandom.java:468) java.math.BigInteger.randomBits(BigInteger.java:634) java.math.BigInteger.(BigInteger.java:623) org.bouncycastle.util.BigIntegers.createRandomInRange(Unknown Source) org.bouncycastle.math.internal.Primes.enhancedMRProbablePrimeTest(Unknown Source) org.bouncycastle.crypto.asymmetric.KeyUtils.validatedModulus(Unknown Source) org.bouncycastle.crypto.asymmetric.KeyUtils.validated(Unknown Source) org.bouncycastle.crypto.asymmetric.AsymmetricRSAPublicKey.(Unknown Source) org.bouncycastle.jcajce.provider.ProvRSAPublicKey.(Unknown Source) org.bouncycastle.jcajce.provider.ProvRSA$1.convertKey(Unknown Source) org.bouncycastle.jcajce.provider.ProvRSA$1.convertKey(Unknown Source) org.bouncycastle.jcajce.provider.BaseSingleBlockCipher.engineInit(Unknown Source) org.bouncycastle.jcajce.provider.BaseSingleBlockCipher.engineInit(Unknown Source) javax.crypto.Cipher.init(Cipher.java:1245) javax.crypto.Cipher.init(Cipher.java:1185) com.glide.util.RSAEncrypter.encryptBytes(RSAEncrypter.java:54) com.glideapp.ecc.MIDServerInfoPayloadEncrypter$KeyPayload.(MIDServerInfoPayloadEncrypter.java:75) com.glideapp.ecc.MIDServerInfoPayloadEncrypter.encryptPayload(MIDServerInfoPayloadEncrypter.java:46) com.glide.script.MIDServerInfoAccessor.loadCredentials(MIDServerInfoAccessor.java:569) com.glide.script.MIDServerInfoAccessor.getCredentialXML(MIDServerInfoAccessor.java:137) sun.reflect.GeneratedMethodAccessor510.invoke(Unknown Source)... Root cause in this example: Further investigation of the code lines seen in the stack trace revealed that the FileInputStream we were reading was actually the Linux /dev/random device. We were waiting for a random number to use for the encryption of the SOAP payload. Sensitive information like credentials are not just encrypted at the TLS level, but super-encrypted before sending in SOAP response. The Pool of available random number for jobs like this is called "Entropy", and is the responsibility of the Linux OS, rather than the ServiceNow application. Our code was blocked, waiting for the next available random number from Linux. This script will print out the current Available Entropy for the Linux host server: # Script to print out entropy count once/ per secondfor i in $(seq 60 $END); do date; cat /proc/sys/kernel/random/entropy_avail; sleep 1; done; This script can be used to increase the Pool Count, increasing the level of entropy available at any time. # Script to increase entropy pool count (runs forever, so ^C out after a few seconds or a minute). dd if=/dev/sda of=/dev/zero count=10000000 If the server the app nodes is running on is on-premise, rather than in servicenow's datacenter, then this value may have been configured too low for the ServiceNow application, and would need raising. Related LinksOnce of the main causes of MID Servers to fail to fetch credentials is broken records in the credential table after a clone. This tends to cause the "SEVERE *** ERROR *** An error occurred while decrypting credentials from instance" error instead, even though the above web service does not itself error. (see PRB1508497 GetMIDInfo SOAP service reports result code 200 even when it gets drastic errors and has no results to return) At the time of writing, this is an open defect:PRB1542851 A clone can corrupt the discovery_credentials table on the target instance, leaving orphan/ghost records with a class that no longer exists, preventing MID Server using all credentials This similar cause is fixed, specifically for the credential table, but other TPC extended tables still have this issue.PRB1305469 Excluding table-per-class (TPC) extended tables from a clone can cause orphaned Discovery Credentials with the 'Record not found' error when trying to open them Known problems that can cause long running /GetMIDInfo.do requests include: PRB1665907 KB1369852 /GetMIDInfo.do (CloudServiceAccountInfoUtil) hangs in an infinite loop until it exhausts node memory or times out, when cmdb_ci_cloud_service_account records have empty account_idPRB1567004 KB0962403 MID Server Service Startup Error "redirect count exceeded or request did not finish within 60 seconds"PRB1401009 Long delay for Linux MID Server to start when Linux OS random number entropy pool runs low (fixed before release)PRB1576909 KB1120158 Memory and/or API_INT semaphore exhaustion due to /GetMIDInfo.do requests when cloud_service_account_view records have empty sa_account_id valuePRB1505334 KB0964591 Post Cloud Resource or GCP Discovery where thousands of cloud service accounts records are updated, all MID servers connected to the instance may show as DOWN and keep throwing "GetCloudServiceAccountInfo-request failed" error messages.