MID Servers and IP Address Access Control


Description

The Paris version documentation for IP Address Access Control is a bit incomplete in relation to this feature's potential effect on MID Servers. This KB aims to fill the gaps until that page is updated.

Navigate to System Security > IP Address Access Control to see a list of your IP access controls. You might have to activate the IP Range Based Authentication [com.snc.ipauthenticator] plugin on older instances to use this feature

MID Server related fields

The following fields on the IP access controls form are not covered by the Paris documentation:

What if an IP the MID Server needs to communicate with is blocked for outbound from the Instance + MID Server

Any probe is affected, for any feature, including Discovery, REST/SOAP, Orchestration, IntergrationsHub etc.

LDAP server example:

I set up an LDAPS server with IP 10.253.253.253, set to communicate through the MID Server. Here is the Outbound rule that blocks that IP, and the LDAP and LDAPS ports 389 and 636.

Here is the LDAP Server record in the instance. This "Verify server address and port are correct and accessible" is the error that you see at the top of the page if you click the 'Test Connection' UI action, which gives no clue that IP Access Control is what is blocking it.

This is the ecc_queue input record for the LDAPConnectionTesterProbe, showing LDAP Error Code "ERR_CODE=10300,ERR_MSG=10.253.253.253:636" in the payload, which is again a generic communication error.

This is what the LDAPConnectionTesterProbe looks like in the MID Server agent log if the IP is blocked, and confirms "OUTBOUND IP BLOCKED". The sys_id in the thread name is the output ecc_queue record, referenced by the Response To field above:

11/11/20 00:16:19 (585) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 Worker starting: LDAPConnectionTesterProbe source: eecd75a30a0a0b2600791193785025b2
11/11/20 00:16:20 (391) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 WARNING *** WARNING *** OUTBOUND IP BLOCKED: 10.253.253.253:636 not authorized
11/11/20 00:16:20 (391) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 LDAP API - LDAPLogger : 10.253.253.253:636
11/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 LDAP API - LDAPLogger : Communication error: 10.253.253.253:636
11/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 LDAP API - LDAPLogger : java.lang.SecurityException: 10.253.253.253:636 not authorized
11/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 Enqueuing: C:\ServiceNow_MID_Servers\dev\agent\work\monitors\ECCSender\output_s\ecc_queue.175b66085ae0000002.xml
11/11/20 00:16:20 (398) Worker-Expedited:LDAPConnectionTesterProbe-3d30ab531bd068507e63415bbc4bcb53 Worker completed: LDAPConnectionTesterProbe source: eecd75a30a0a0b2600791193785025b2 time: 0:00:00.813

The following is what the LDAP Listener running on the MID Server looks like in the MID Server agent log if the IP is blocked:

11/11/20 00:27:40 (427) Worker-Standard:LDAPListenProbe-77c26f971bd4e450254542e7cc4bcb3b Worker starting: LDAPListenProbe source: eecd75a30a0a0b2600791193785025b2
11/11/20 00:27:40 (443) ECCQueueMonitor.1 DEBUG: HTTPClient.registerOtherProtocols() starting on Thread Thread[ECCQueueMonitor.1,5,main].
11/11/20 00:27:40 (614) Worker-Standard:LDAPListenProbe-77c26f971bd4e450254542e7cc4bcb3b Enqueuing: C:\ServiceNow_MID_Servers\dev\agent\work\monitors\ECCSender\output_s\ecc_queue.175b66ae6c60000001.xml
11/11/20 00:27:40 (614) Worker-Standard:LDAPListenProbe-77c26f971bd4e450254542e7cc4bcb3b Worker completed: LDAPListenProbe source: eecd75a30a0a0b2600791193785025b2 time: 0:00:00.187
11/11/20 00:27:40 (631) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 Getting instance ACLs for table: sys_status
11/11/20 00:27:42 (098) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 Getting instance ACLs for table: ldap_ou_config
11/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : 10.253.253.253:636
11/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : Communication error: 10.253.253.253:636
11/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : java.lang.SecurityException: 10.253.253.253:636 not authorized
11/11/20 00:27:43 (707) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 WARNING *** WARNING *** LDAP API - LDAPLogger : Connection error.  Waiting 1 seconds to retry
11/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : 10.253.253.253:636
11/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : Communication error: 10.253.253.253:636
11/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 LDAP API - LDAPLogger : java.lang.SecurityException: 10.253.253.253:636 not authorized
11/11/20 00:27:47 (348) glide.ldap.listener-eecd75a30a0a0b2600791193785025b2 WARNING *** WARNING *** LDAP API - LDAPLogger : Connection error.  Waiting 2 seconds to retry
...

What if the MID Server's IP is blocked for inbound to the Instance?

By default, there are no restrictions on access to your instance. If you have authentication and 403 errors in the MID Server logs, then this is usually one of the last things you need to check. First check if any IP Address Access Control rules are set in the instance at all, to discount this being relevant.

However if a rule like this existed, and the MID Server host (or the proxy it connects through) has that IP, the MID Server would go Down.

Example:

Here is a MID Server record. Note the reported IP of the host is 10.0.25.15. That is an internal/unrouteable IP, and so it can't be the IP that requests to the instance actually come from.

Here is what an instance app node localhost looks like when a MID Server is able to communicate normally. This is one way to confirm the IP that requests from the MID Server are actually coming from, and that the MID Server is communicating normally. If you don't see logs of SOAP requests like that for MID Server related tables, for the MID Server user, then the MID Server requests are not getting to the instance.

2020-11-10 00:00:35 (714) API_INT-thread-4 B66F8A0B1B90A4507E63415BBC4BCB43 txid=a5031ac71b90 *** End  #24962 /ecc_queue.do, user: mid_user, total time: 0:00:00.167, processing time: 0:00:00.167, SQL time: 0:00:00.137 (count: 8), ACL time: 0:00:00.001, source: 94.208.87.250 , type:soap, method:getRecords, api_name:SOAP APIs, resource:ecc_queue.do, user_id:e66974abdb813300e1943ecf9d9619d8, response_status:200 

That shows a MID server, logging in as user "mid_user", which is the mid_server role user, accessing the ecc_queue table either to fetch or return job results, from source IP 94.208.87.250.

That can also be confirmed from syslog_transaction, as an Admin user:

A rule was created to block that IP:

The MID Server would go Down, and you would see these "Forbidden with code: 403" errors in the MID Server agent log. This example shows the moment it gets blocked, where the MID Server was able to fetch the SystemCommand job for 'load_ip_access' and run it, but then could not send the result back:

2020-11-10 07:49:36  (173) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Worker starting: SystemCommand source: load_ip_access
2020-11-10 07:49:36  (173) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Running system command: load_ip_access
2020-11-10 07:49:36  (405) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 WARNING *** WARNING *** Method failed: (https://empdpiper.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403
2020-11-10 07:49:36  (405) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://empdpiper.service-now.com/ with (Method failed: (https://empdpiper.service-now.com/ecc_agent_log.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403)
2020-11-10 07:49:36  (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 WARNING *** WARNING *** Method failed: (https://empdpiper.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403
2020-11-10 07:49:36  (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 SEVERE *** ERROR *** getRecords failed (Method failed: (https://empdpiper.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403)
2020-11-10 07:49:36  (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 SEVERE *** ERROR *** Failed to execute remote query: Method failed: (https://empdpiper.service-now.com/ip_access.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403
2020-11-10 07:49:36  (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Enqueuing: C:\ServiceNow_MID_Servers\dev\agent\work\monitors\ECCSender\output_0\ecc_queue.675ebf4b1b1028507e63415bbc4bcb67.xml
2020-11-10 07:49:36  (515) Worker-Interactive:SystemCommand-675ebf4b1b1028507e63415bbc4bcb67 Worker completed: SystemCommand source: load_ip_access time: 0:00:00.342
2020-11-10 07:49:36  (843) ECCSender.1 Sending ecc_queue.675ebf4b1b1028507e63415bbc4bcb67.xml
2020-11-10 07:49:37  (001) ECCSender.1 WARNING *** WARNING *** Method failed: (https://empdpiper.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403
2020-11-10 07:49:37  (001) ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://empdpiper.service-now.com/ with (Method failed: (https://empdpiper.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 403 Forbidden with code: 403)
2020-11-10 07:49:37  (001) ECCSender.1 Attempt to send ecc_queue.675ebf4b1b1028507e63415bbc4bcb67.xml failed: file remains enqueued for later sending

And these warnings in instance app node localhost logs:

2020-11-10 07:49:36 (188) http-5 WARNING *** WARNING *** Security restricted: Access restricted (94.208.87.250 not authorized)

Opening the instance URL in a Browser on the MID Server's host will also have the same 403 error, but provide a bit more information on this specific IP being not authorized: