MID Server initial connection to instance shows 403 Forbidden in logs for on-premise instanceIssue <!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } A MID Server is not connecting to an on-premise instance, and seeing a '403 Forbidden' response during initial connection. The error occurs during the first request to GetMIDInfo, with logs (wrongly) indicating authentication issues for the mid_server role user. Symptoms<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Instance is on-premise, configured by the customer.MID Server user roles and password will already have been confirmed to be correct, and Access Analyser will pass for the various MID Server tables, such as ecc_agent.All MID Server related scripted SOAP Services will be confirmed OOTB.The request to /GetMIDInfo.do is NOT see in instance appnode localhost logs, indicating it doesn't get as far as the app node, meaning something blocked it before it got there.The response body is in HTML, which is not what you'd normally expect for a User authentication or API ACL failure, again suggesting it isn't the instance giving the error.However some other requests that the MID Server makes on startup might be getting through to the appnodesm and seen in the syslog_transaction table, such as /ecc_agent_isse.do, /ldap_server_config.do, /file_discovery_agent_sync.do. That proves the user is authenticated.Other forms and pages in the instance may also be returning a page displaying "403 Forbidden", suggesting the MID Server might not be the only thing affected. MID Server agent0.log.0 shows the following for the initial request to the /GetMIDInfo.do scripted SOAP service request during MID Server startup. ... ERROR (StartupSequencer) [InstanceSOAPClient:139] SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><roles xsi:type="xsd:string">mid_server,soap,soap_script,soap_query,soap_create,soap_delete,soap_ecc,soap_script,soap_update</roles></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>... ERROR (StartupSequencer) [InstanceSOAPClient:139] SOAP Response: Status code=403, Response body=<html><head><title>403 Forbidden</title></head><body><center><h1>403 Forbidden</h1></center></body></html> Subsequent error text is misleading. All of those suggestions can easily be have been discounted as the cause. ... WARN (StartupSequencer) [UserConfigTest:44] Could not check roles on instance: null... ERROR (StartupSequencer) [StartupSequencer:712] test failurejava.lang.IllegalStateException: User: mid_server_pre_prod cannot be authenticated. User does not exist, or missing the proper roles. If you have deleted or changed the MID server keystore, and config.xml mid.instance.password value is encrypted, you may need to change this value to plain text (during MID startup, password is re-encrypted using current keystore and written back to mid.instance.password).at com.service_now.mid.services.StartupSequencer.runTests(StartupSequencer.java:650)at com.service_now.mid.services.StartupSequencer.startupSequencerRunnable(StartupSequencer.java:709)at java.base/java.lang.Thread.run(Thread.java:840) ... Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Unrelated to instance version. Cause<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Customer's Web Application Firewall (WAF) rules were found to be the cause. This may be implemented in the instance's Load Balancer, or a separate appliance. More information on WAF in general can be found in: Wikipedia: Web application firewall Docs: Security Operations - Security Posture Control - Exploit Protection (WAF) mitigation controls Amazon: AWS WAF Rules Resolution<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Test and resolve the issues with the WAF Rules.