MID Servers and IP Address Access ControlSummary Table of Contents MID Server related fieldsHow the rules are synchronisedKnown issues and Debugging information What if an IP the MID Server needs to communicate with is blocked for outbound from the Instance + MID ServerWhat if the MID Server's IP is blocked for inbound to the Instance?MID Servers Shutting themselves down when the ip_access table isn't accessibleMID Servers Shutting themselves down when the ip_access table isn't there (Plugin not installed) The documentation for IP Address Access Control is a bit incomplete in relation to this feature's potential effect on MID Servers. This KB aims to fill the gaps until that page is updated. Navigate to System Security > IP Address Access Control to see a list of your IP access controls. You might have to activate the IP Range Based Authentication [com.snc.ipauthenticator] plugin on older instances to use this feature MID Server related fields The following fields on the IP access controls form are not covered by the Paris documentation: Enforce on MID server When checked, the rule will apply for requests sent by both the MID Server and the instance. You cannot define a MID Server only rule, that does not also apply to the instance.This applied to outbound requests, via the MID Server. That includes all probes. This does not apply to inbound requests to the MID Server Web Server Extension Specify ports If "Specify ports" is selected, the rule will apply to specific ports.Enter a comma-separated list of individual ports or ranges (e.g., 12,20-25,8080). Empty indicates all ports, so you have to enter port numbers for this to have any effect on the rule.Ports only apply to Outbound direction rules. If Bidirectional direction is selected, port rules only apply to the outbound connections. How the rules are synchronised These rules are stored in the ip_access table in the instance, and where the Enforce on MID server field is ticked, are synchronized to MID Servers: On startup, when the StartupSequencer thread initializes the IPAccessConfig class.Since Rome, Quebec Patch 6, Paris Patch 10 By clicking the "Sync rules to MID" related link under the IP Address Access Controls list. A "load_ip_access" SystemCommand is put in the ecc_queue for all MID Servers.By clicking the "Restart MID" related link on a MID Server form. As well as a SystemCommand to restart the MID Server, a "load_ip_access" SystemCommand is put in the ecc_queue for this MID Server. Before Rome, Quebec Patch 6, Paris Patch 10 By updating a ip_access record, which triggered a DB Listener. A "load_ip_access" SystemCommand is put in the ecc_queue for this MID Server by that DB listener, for each record updated. Mass updates to that table by imports, would cause such a flood of system commands that it caused MID Server outages (PRB1499288, hence the change in behaviour). Note: The fact the "Restart MID" creates "load_ip_access" SystemCommand, even though the instance does not have the IP Range Based Authentication plugin, and ip_access table installed, might be causing problems - TBC. Known issues and Debugging information What if an IP the MID Server needs to communicate with is blocked for outbound from the Instance + MID Server Any probe is affected, for any feature, including Discovery, REST/SOAP, Orchestration, IntergrationsHub etc. LDAP server example: I set up an LDAPS server with IP 10.253.253.253, set to communicate through the MID Server. Here is the Outbound rule that blocks that IP, and the LDAP and LDAPS ports 389 and 636. Here is the LDAP Server record in the instance. This "Verify server address and port are correct and accessible" is the error that you see at the top of the page if you click the 'Test Connection' UI action, which gives no clue that IP Access Control is what is blocking it. This is the ecc_queue input record for the LDAPConnectionTesterProbe, showing LDAP Error Code "ERR_CODE=10300,ERR_MSG=10.253.253.253:636" in the payload, which is again a generic communication error. This is what the LDAPConnectionTesterProbe looks like in the MID Server agent log if the IP is blocked, and confirms "OUTBOUND IP BLOCKED". The sys_id in the thread name is the output ecc_queue record, referenced by the Response To field above: 11/11/20 00:16:19 (585) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 Worker starting: LDAPConnectionTesterProbe source: eecd75a30a0a0b2600791193785025b211/11/20 00:16:20 (391) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 WARNING *** WARNING *** OUTBOUND IP BLOCKED: 10.253.253.253:636 not authorized11/11/20 00:16:20 (391) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 LDAP API - LDAPLogger : 10.253.253.253:63611/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 LDAP API - LDAPLogger : Communication error: 10.253.253.253:63611/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 LDAP API - LDAPLogger : java.lang.SecurityException: 10.253.253.253:636 not authorized11/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 Enqueuing: C:\ServiceNow_MID_Servers\dev\agent\work\monitors\ECCSender\output_s\ecc_queue.175b66085ae0000002.xml11/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 Worker completed: LDAPConnectionTesterProbe source: eecd75a30a0a0b2600791193785025b2 time: 0:00:00.813 The following is what the LDAP Listener running on the MID Server looks like in the MID Server agent log if the IP is blocked: 11/11/20 00:27:40 (427) Worker-Standard:LDAPListenProbe-77c26f971bd4e450254542e7cc4bcb3b Worker starting: LDAPListenProbe source: eecd75a30a0a0b2600791193785025b211/11/20 00:27:40 (443) ECCQueueMonitor.1 DEBUG: HTTPClient.registerOtherProtocols() starting on Thread Thread[ECCQueueMonitor.1,5,main].11/11/20 00:27:40 (614) Worker-Standard:LDAPListenProbe-77c26f971bd4e450254542e7cc4bcb3b Enqueuing: C:\ServiceNow_MID_Servers\dev\agent\work\monitors\ECCSender\output_s\ecc_queue.175b66ae6c60000001.xml11/11/20 00:27:40 (614) Worker-Standard:LDAPListenProbe-77c26f971bd4e450254542e7cc4bcb3b Worker completed: LDAPListenProbe source: eecd75a30a0a0b2600791193785025b2 time: 0:00:00.18711/11/20 00:27:40 (631) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 Getting instance ACLs for table: sys_status11/11/20 00:27:42 (098) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 Getting instance ACLs for table: ldap_ou_config11/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : 10.253.253.253:63611/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : Communication error: 10.253.253.253:63611/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : java.lang.SecurityException: 10.253.253.253:636 not authorized11/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 WARNING *** WARNING *** LDAP API - LDAPLogger : Connection error. Waiting 1 seconds to retry11/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : 10.253.253.253:63611/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : Communication error: 10.253.253.253:63611/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : java.lang.SecurityException: 10.253.253.253:636 not authorized11/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 WARNING *** WARNING *** LDAP API - LDAPLogger : Connection error. Waiting 2 seconds to retry... What if the MID Server's IP is blocked for inbound to the Instance? By default, there are no restrictions on access to your instance. If you have authentication and 403 errors in the MID Server logs, then this is usually one of the last things you need to check. First check if any IP Address Access Control rules are set in the instance at all, to discount this being relevant. However if a rule like this existed, and the MID Server host (or the proxy it connects through) has that IP, the MID Server would go Down. Example: Here is a MID Server record. Note the reported IP of the host is 10.0.25.15. That is an internal/unrouteable IP, and so it can't be the IP that requests to the instance actually come from. Here is what an instance app node localhost looks like when a MID Server is able to communicate normally. This is one way to confirm the IP that requests from the MID Server are actually coming from, and that the MID Server is communicating normally. If you don't see logs of SOAP requests like that for MID Server related tables, for the MID Server user, then the MID Server requests are not getting to the instance. 2020-11-10 00:00:35 (714) API_INT-thread-4 B66F8A0B1B90A4507E63415BBC4BCB43 txid=a5031ac71b90 *** End #24962 /ecc_queue.do, user: mid_user, total time: 0:00:00.167, processing time: 0:00:00.167, SQL time: 0:00:00.137 (count: 8), ACL time: 0:00:00.001, source: 94.208.87.250 , type:soap, method:getRecords, api_name:SOAP APIs, resource:ecc_queue.do, user_id:e66974abdb813300e1943ecf9d9619d8, response_status:200 That shows a MID server, logging in as user "mid_user", which is the mid_server role user, accessing the ecc_queue table either to fetch or return job results, from source IP 94.208.87.250. That can also be confirmed from syslog_transaction, as an Admin user: A rule was created to block that IP: The MID Server would go Down, and you would see these "Forbidden with code: 403" errors in the MID Server agent log. This example shows the moment it gets blocked, where the MID Server was able to fetch the SystemCommand job for 'load_ip_access' and run it, but then could not send the result back: 2020-11-10 07:49:36 (173) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Worker starting: SystemCommand source: load_ip_access2020-11-10 07:49:36 (173) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Running system command: load_ip_access2020-11-10 07:49:36 (405) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 WARNING *** WARNING *** Method failed: (https://empdpiper.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 4032020-11-10 07:49:36 (405) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://empdpiper.service-now.com/ with (Method failed: (https://empdpiper.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403)2020-11-10 07:49:36 (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 WARNING *** WARNING *** Method failed: (https://empdpiper.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 4032020-11-10 07:49:36 (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 SEVERE *** ERROR *** getRecords failed (Method failed: (https://empdpiper.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403)2020-11-10 07:49:36 (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 SEVERE *** ERROR *** Failed to execute remote query: Method failed: (https://empdpiper.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 4032020-11-10 07:49:36 (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Enqueuing: C:\ServiceNow_MID_Servers\dev\agent\work\monitors\ECCSender\output_0\ecc_queue.675ebf4b1b1028507e63415bbc4bcb67.xml2020-11-10 07:49:36 (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Worker completed: SystemCommand source: load_ip_access time: 0:00:00.3422020-11-10 07:49:36 (843) ECCSender.1 Sending ecc_queue.675ebf4b1b1028507e63415bbc4bcb67.xml2020-11-10 07:49:37 (001) ECCSender.1 WARNING *** WARNING *** Method failed: (https://empdpiper.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 4032020-11-10 07:49:37 (001) ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://empdpiper.service-now.com/ with (Method failed: (https://empdpiper.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403)2020-11-10 07:49:37 (001) ECCSender.1 Attempt to send ecc_queue.675ebf4b1b1028507e63415bbc4bcb67.xml failed: file remains enqueued for later sending And these warnings in instance app node localhost logs: 2020-11-10 07:49:36 (188) http-5 WARNING *** WARNING *** Security restricted: Access restricted (94.208.87.250 not authorized) Opening the instance URL in a Browser on the MID Server's host will also have the same 403 error, but provide a bit more information on this specific IP being not authorized: MID Servers Shutting themselves down when the ip_access table isn't accessible If ACL rules/roles mean the MID Server logon user is unable to read the ip_access table or records, a "403" error is shown in the agent log, followed by a shutdown. StartupSequencer SEVERE *** ERROR *** getRecords failed (Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403) or INFO (StartupSequencer) [SOAPSecurity:59] Getting instance ACLs for table: ip_accessWARN (StartupSequencer) [HTTPClient:830] Method failed: (https://xxx.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500ERROR (StartupSequencer) [RemoteGlideRecord:928] getRecords failed (com.glide.processors.soap.SOAPProcessingException: Field(s) present in the query do not have permission to be read com.glide.processors.soap.SOAPProcessingException: Field(s) present in the query do not have permission to be read) Solution: Check the MID Server user has the mid_server role, and that these 2 ACLs are active, and have the mid_server role linked: record/ip_access/read /sys_security_acl.do?sys_id=6296ab420f201010a86b657eef767e59record/ip_access.*/read /sys_security_acl.do?sys_id=7ab62f420f201010a86b657eef767e5a These 2 ACLs were added in Paris, as the fix for PRB1399614. MID Servers Shutting themselves down when the ip_access table isn't there (Plugin not installed) Note: PRB1687284 now exists for this. Not all instances have the IP Range Based Authentication [com.snc.ipauthenticator] plugin installed. That will mean the ip_access table is not there. If the MID Server does try to sync the IP Access rules from that instance it will fail, and it then shuts itself down, and restart. That restart loop will continue until the access to ip_access table is available. 2022-02-18 20:19:29 (021) StartupSequencer WARNING *** WARNING *** Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 5002022-02-18 20:19:29 (021) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access)2022-02-18 20:19:29 (021) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 10 seconds...2022-02-18 20:28:51 (592) StartupSequencer WARNING *** WARNING *** Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 5002022-02-18 20:28:51 (592) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access)2022-02-18 20:28:51 (592) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 120 seconds2022-02-18 20:30:51 (661) StartupSequencer WARNING *** WARNING *** Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 5002022-02-18 20:30:51 (661) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access)2022-02-18 20:30:51 (661) StartupSequencer SEVERE *** ERROR *** Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500com.service_now.mid.MIDRemoteGlideRecord$RetryableException: Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500 at com.service_now.mid.MIDRemoteGlideRecord.handleResult(MIDRemoteGlideRecord.java:140) at com.service_now.mid.MIDRemoteGlideRecord.lambda$query$2(MIDRemoteGlideRecord.java:89) at com.service_now.mid.util.RetryExecutor.execute(RetryExecutor.java:84) at com.service_now.mid.MIDRemoteGlideRecord.executeWithRetry(MIDRemoteGlideRecord.java:172) at com.service_now.mid.MIDRemoteGlideRecord.executeBooleanWithRetry(MIDRemoteGlideRecord.java:183) at com.service_now.mid.MIDRemoteGlideRecord.query(MIDRemoteGlideRecord.java:93) at com.service_now.mid.services.IPAccessConfig.loadRemoteIpAccessRules(IPAccessConfig.java:56) at com.service_now.mid.services.IPAccessConfig.init(IPAccessConfig.java:36) at com.service_now.mid.services.StartupSequencer.startServices(StartupSequencer.java:366) at com.service_now.mid.services.StartupSequencer.testsSucceeded(StartupSequencer.java:170) at com.service_now.mid.services.StartupSequencer.startupSequencerRunnable(StartupSequencer.java:730) at java.base/java.lang.Thread.run(Thread.java:834)2022-02-18 20:30:51 (661) StartupSequencer SEVERE *** ERROR *** Failed to execute remote query: Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500...2022-02-18 20:31:22 (373) Wrapper-Restarter Using configuration: E:\ServiceNow\Prod\Discover\config.xml2022-02-18 20:31:22 (420) Wrapper-Restarter Logger config: root=INFO2022-02-18 20:31:22 (420) Wrapper-Restarter Refreshing LoggerFactory cache2022-02-18 20:31:22 (436) Wrapper-Restarter Stopping MID server2022-02-18 20:31:22 (436) Wrapper-Restarter Main.handleStop() before shutdown, OperationalState=UP2022-02-18 20:31:22 (482) Wrapper-Restarter Setting mid status to Down... There is also an equivalent error in the instance app node logs. The "web service" error is misleading. If no table exists, the SOAP interface assumes it's probably a request to a web service, rather than the SOAP table API. 2022-02-18 23:02:02 (054) API_INT-thread-1 ... *** Start #96389 /ip_access.do, user: midserver2022-02-18 23:02:02 (065) API_INT-thread-1 ... WARNING *** WARNING *** SOAP Fault: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode>SOAP-ENV:Server</faultcode><faultstring>com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access</faultstring><detail>com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access</detail></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope> It can also look like this: 2022-02-18 20:42:25 (058) StartupSequencer WARNING *** WARNING *** Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 5002022-02-18 20:42:25 (058) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access)2022-02-18 20:42:25 (058) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 10 seconds...2022-02-18 20:47:47 (198) StartupSequencer WARNING *** WARNING *** Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 5002022-02-18 20:47:47 (198) StartupSequencer SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access)2022-02-18 20:47:47 (198) StartupSequencer WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 120 seconds2022-02-18 20:49:44 (936) WrapperListener_stop_runner Using configuration: E:\ServiceNow\Prod\Discover\config.xml2022-02-18 20:49:45 (015) WrapperListener_stop_runner Logger config: root=INFO2022-02-18 20:49:45 (015) WrapperListener_stop_runner Refreshing LoggerFactory cache2022-02-18 20:49:45 (015) WrapperListener_stop_runner Stopping MID server2022-02-18 20:49:45 (015) WrapperListener_stop_runner Main.handleStop() before shutdown, OperationalState=UP2022-02-18 20:49:45 (077) WrapperListener_stop_runner Setting mid status to Down... When "Restart MID" is clicked on more recent versions, a "load_ip_access" SystemCommand is sent to the MID Server. When this runs, the agent log looks like: 01/22/22 08:02:23 (954) Worker-Interactive:SystemCommand-07bccb801b91411063a3a642b24bcb96 WARNING *** WARNING *** Method failed: (https://<instance>.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 50001/22/22 08:02:23 (954) Worker-Interactive:SystemCommand-07bccb801b91411063a3a642b24bcb96 SEVERE *** ERROR *** getRecords failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access com.glide.processors.soap.SOAPProcessingException: Web service not found: ip_access)01/22/22 08:02:23 (969) Worker-Interactive:SystemCommand-07bccb801b91411063a3a642b24bcb96 WARNING *** WARNING *** MIDRemoteGlideRecord.query failed, retrying in 15 seconds...01/22/22 08:02:24 (047) Wrapper-Restarter Setting mid status to Down Solution: For "500" errors, enable the IP Range Based Authentication [com.snc.ipauthenticator] plugin. It will add the ip_access table, which will be empty by default. You do not need to configure any rules, or use the feature, to workaround this issue.