An error occurred while decrypting credentials from instance - Creating OAuth 2.0 credential results in no credentials being retrieved by the MID server and Probes no longer being able to use them


Description

If there is an OAuth 2.0 type Credential [oauth_2_0_credentials] added to the list of Credentials in the instance, the instance will no longer return any credentials to the MID server when the MID Server tries to re-load the credential list. The MID Server will no longer be able to run any probes that requires credentials. This includes all Discovery, Orchestration, Event Management Connector probes, and others.

The MID Server agent logs will show a "SEVERE *** ERROR *** An error occurred while decrypting credentials from instance" when running each affected probe.

Steps to Reproduce

  1. Create a new OAuth 2.0 credential using an OOB OAuth Entity Profile (e.g. Google default_profile) on an instance with at least one MID running.
  2. Verify that an ECC queue output record has been processed by the MID(s) with the topic credentials_reload. The agent logs will also confirm this ran.
  3. Verify that in the agent log file for the MID(s) there is an exception that looks like the following. This is the exception thrown when no key element is found in the response (i.e., the response is empty)
SEVERE *** ERROR *** An error occurred while decrypting credentials from instance
com.snc.automation_common.integration.exceptions.AutomationIOException: Unable to retrieve data from instance. This MID may not be validated. <<<=== Red Herring - The MID Server is Validated.
at com.glide.util.MIDServerInfoPayloadDecrypter.decryptPayload(MIDServerInfoPayloadDecrypter.java:25)
at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.loadCredentials(StandardCredentialsProvider.java:289)
at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.load(StandardCredentialsProvider.java:256)
at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.init(StandardCredentialsProvider.java:275)
at com.service_now.mid.creds.provider.MIDCredentialsConfigProvider.getCredentialsProvider(MIDCredentialsConfigProvider.java:58)
at com.snc.commons.credentials.CredentialsProviderFactory.getCredentialsProvider(CredentialsProviderFactory.java:30)
...

Lines below this in the stack trace will be specific to the probe running.

Workaround

This problem has been fixed. There is no workaround available. If you are able to upgrade, review the Fixed In section to determine the latest version with a permanent fix your instance can be upgraded to.

The workaround is to ensure the affected MID Servers do not have any OAuth Credentials set up for them:

NOTE: The same symptom/error could also be due to PRB1305469 Excluding table-per-class (TPC) extended tables from a clone can cause orphaned Discovery Credentials with the 'Record not found' error when trying to open them

If you also have thousands of source=credentials_reload jobs backed-up in the ECC Queue, then you probably also experiencing this, which requires additional steps to resolve:
PRB1411442 / KB0829702 OAuth 2.0 credential that is configured with extremely short TTL (<1 minute) causes ECC queue flooding (credentials_reload command) and leads to semaphore exhaustion on instance


Related Problem: PRB1342894