If the IP Range Based Authentication [com.snc.ipauthenticator] plugin is not installed, the missing ip_access table causes the MID Server to go Down - Known Error

If the IP Range Based Authentication [com.snc.ipauthenticator] plugin is not installed, the missing ip_access table causes the MID Server to go Down Description Older instances may not have the IP Range Based Authentication [com.snc.ipauthenticator] plugin installed. This can cause the MID Server to get into a restart loop, and be effectively down, due to the required ip_access table being missing in the instance.

Steps to Reproduce

On an instance that does not have the IP Range Based Authentication [com.snc.ipauthenticator] plugin installed. This will mean having quite an old instance. Install a MID Server, and then click the "Restart MID" UI action on its form Since Rome, Quebec Patch 6, Paris Patch 10 (PRB1499288 fix), as well as a SystemCommand to restart the MID Server, a "load_ip_access" SystemCommand is put in the ecc_queue for this MID Server to re-sync the IP Access Control rules. Expected behaviour: The MID should just restart. IP Range Based Authentication [com.snc.ipauthenticator] plugin should be a dependency of the MID Server plugin. Just installing the plugin doesn't enable any rules, but does add the table that the MID Server needs.

Actual behaviour: The MID Server goes into a restart loop, in effect making it Down and unusable. Note: ip_access is not a Web Service; it is a table. When a table is not there the soap table API assumes it might be a scripted web service instead, hence the "Web service not found" error.

2022-02-18 20:19:29 (021) StartupSequencer WARNING *** WARNING *** Method failed: (https:// .service-now.com/ip_access.do ?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 2022-02-18 20:19:29 (021) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access ) 2022-02-18 20:19:29 (021) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 10 seconds ... 2022-02-18 20:28:51 (592) StartupSequencer WARNING *** WARNING *** Method failed: (https:// .service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 2022-02-18 20:28:51 (592) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access) 2022-02-18 20:28:51 (592) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 120 seconds 2022-02-18 20:30:51 (661) StartupSequencer WARNING *** WARNING *** Method failed: (https:// .service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 2022-02-18 20:30:51 (661) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access) 2022-02-18 20:30:51 (661) StartupSequencer SEVERE *** ERROR *** Method failed: (https:// .service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 com.service_now.mid.MIDRemoteGlideRecord$RetryableException: Method failed: (https:// .service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 at com.service_now.mid.MIDRemoteGlideRecord.handleResult(MIDRemoteGlideRecord.java:140) at com.service_now.mid.MIDRemoteGlideRecord.lambda$query$2(MIDRemoteGlideRecord.java:89) at com.service_now.mid.util.RetryExecutor.execute(RetryExecutor.java:84) at com.service_now.mid.MIDRemoteGlideRecord.executeWithRetry(MIDRemoteGlideRecord.java:172) at com.service_now.mid.MIDRemoteGlideRecord.executeBooleanWithRetry(MIDRemoteGlideRecord.java:183) at com.service_now.mid.MIDRemoteGlideRecord.query(MIDRemoteGlideRecord.java:93) at com.service_now.mid.services. IPAccessConfig.loadRemoteIpAccessRules(IPAccessConfig.java:56) at com.service_now.mid.services.IPAccessConfig.init(IPAccessConfig.java:36) at com.service_now.mid.services.StartupSequencer.startServices(StartupSequencer.java:366) at com.service_now.mid.services.StartupSequencer.testsSucceeded(StartupSequencer.java:170) at com.service_now.mid.services.StartupSequencer.startupSequencerRunnable(StartupSequencer.java:730) at java.base/java.lang.Thread.run(Thread.java:834) 2022-02-18 20:30:51 (661) StartupSequencer SEVERE *** ERROR *** Failed to execute remote query: Method failed: (https:// .service-now.com/ip_access.do ?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 ... 2022-02-18 20:31:22 (373) Wrapper-Restarter Using configuration: E:\ServiceNow\Prod\Discover\config.xml 2022-02-18 20:31:22 (420) Wrapper-Restarter Logger config: root=INFO 2022-02-18 20:31:22 (420) Wrapper-Restarter Refreshing LoggerFactory cache 2022-02-18 20:31:22 (436) Wrapper-Restarter Stopping MID server 2022-02-18 20:31:22 (436) Wrapper-Restarter Main.handleStop() before shutdown, OperationalState=UP 2022-02-18 20:31:22 (482) Wrapper-Restarter Setting mid status to Down ... There is also an equivalent error in the instance app node logs. The "web service" error is misleading. If no table exists, the SOAP interface assumes it's probably a request to a web service, rather than the SOAP table API.

2022-02-18 23:02:02 (054) API_INT-thread-1 ... *** Start #96389 /ip_access.do, user: midserver 2022-02-18 23:02:02 (065) API_INT-thread-1 ... WARNING *** WARNING *** SOAP Fault: SOAP-ENV:Server com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access It can also look like this:

2022-02-18 20:42:25 (058) StartupSequencer WARNING *** WARNING *** Method failed: (https:// .service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 2022-02-18 20:42:25 (058) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access) 2022-02-18 20:42:25 (058) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 10 seconds ... 2022-02-18 20:47:47 (198) StartupSequencer WARNING *** WARNING *** Method failed: (https:// .service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 2022-02-18 20:47:47 (198) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access) 2022-02-18 20:47:47 (198) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 120 seconds 2022-02-18 20:49:44 (936) WrapperListener_stop_runner Using configuration: E:\ServiceNow\Prod\Discover\config.xml 2022-02-18 20:49:45 (015) WrapperListener_stop_runner Logger config: root=INFO 2022-02-18 20:49:45 (015) WrapperListener_stop_runner Refreshing LoggerFactory cache 2022-02-18 20:49:45 (015) WrapperListener_stop_runner Stopping MID server 2022-02-18 20:49:45 (015) WrapperListener_stop_runner Main.handleStop() before shutdown, OperationalState=UP 2022-02-18 20:49:45 (077) WrapperListener_stop_runner Setting mid status to Down ... When "Restart MID" is clicked on more recent versions, a "load_ip_access" SystemCommand is sent to the MID Server. When this runs, the agent log looks like:

01/22/22 08:02:23 (954) Worker-Interactive:SystemCommand-07bccb801b91411063a3a642b24bcb96 WARNING *** WARNING *** Method failed: (https:// .service-now.com/ip_access.do?SOAP&amp;displayvalue=all&amp;redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 01/22/22 08:02:23 (954) Worker-Interactive:SystemCommand-07bccb801b91411063a3a642b24bcb96 SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access) 01/22/22 08:02:23 (969) Worker-Interactive:SystemCommand-07bccb801b91411063a3a642b24bcb96 WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 15 seconds ... 01/22/22 08:02:24 (047) Wrapper-Restarter Setting mid status to Down Workaround Install the IP Range Based Authentication [com.snc.ipauthenticator] plugin. Just installing the plugin doesn't enable any rules, so doesn't change anything functionally, but does add the table that the MID Server needs.

For more details on the feature, see KB0864024 MID Servers and IP Address Access Control