Agent Client Collectors agents no longer able to connect to ServiceNow Instance through Mid server when using Load balancer address after Australia UpdgradeIssue <!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } After Upgrading from Yokohama to Australia, customers had issues connecting Agents through their F5 Load Balancers to the mid servers Symptoms<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Load Balancers were configured to use TLS Termination: Agent Connects via https to the Load Balancer and then a new session is created to the Mid Server. When they connect directly to the Mid Servers there are no issues. Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Australia Cause<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } In Australia, Jetty's WebSocket handshaker constructs a java.net.URI from the request URL in AbstractHandshaker.upgradeRequest(). Java's java.net.URI host parser is strict per RFC 3986 Section 3.2.2: hostnames may contain letters, digits, hyphens, and dots, but NOT underscores. Browsers, OpenSSL, curl, and most DNS systems are lenient about underscores; java.net.URI is not. Yokohama shipped Jetty 9.4.x where the WebSocket handshake path either avoided java.net.URI or invoked a more lenient parser. Australia ships Jetty 10+ (bumped via STRY60854764). Jetty 10's jetty.websocket.core.server.internal.AbstractHandshaker.upgradeRequest invokes HttpURI.toURI() which builds java.net.URI and throws URISyntaxException on the first underscore in the hostname. The exception propagates up as RuntimeException, the WebSocketUpgradeFilter catches it and returns HTTP 500 to the client. Actual: HTTP/1.1 500 Server Error with response body containing: java.lang.RuntimeException: java.net.URISyntaxException: Illegal character in hostname at index 10: https://<Load Balancer Name with "_" characters in it.example.com:0/ws/events at org.eclipse.jetty.http.HttpURI$Immutable.toURI(HttpURI.java:479) at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker.upgradeRequest(AbstractHandshaker.java:132) at org.eclipse.jetty.websocket.core.server.internal.HandshakerSelector.upgradeRequest at org.eclipse.jetty.websocket.core.server.WebSocketMappings.upgrade at org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter ... Caused by: java.net.URISyntaxException: Illegal character in hostname at index 10 at java.base/java.net.URI$Parser.parseHostname This is an Australia regression. The version of Jetty in Yokohama didn't trigger the strict URI parser on this code path, so the underscored Load Balancer hostname (with "_" in it)may have worked for years. The upgrade to Australia changed the Jetty behavior, and the Load Balancer hostname now fails because it has underscores. The hostname is what's tripping the new strict parser. Resolution<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } 1. Add a DNS record for the load balancer hostname, replacing any underscores with hyphens pointing at the same F5 VIP IP. 2. Verify with curl that the new hostname returns 101 Switching Protocols: curl -v -k --http1.1 \ -H "Connection: Upgrade" \ -H "Upgrade: websocket" \ -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \ -H "Sec-WebSocket-Version: 13" \ -H "Origin: https://<LB Hostname with Hyphens>" \ -H "Authorization: Key " \ https://<LB Hostname with Hyphens>:443/ws/events 3. Update one agent's acc.yml backend-url from wss://<LB Hostname with Underscores>.com:443/ws/events to wss://<LB Hostname with Hyphens>:443/ws/events. Restart that agent. Confirm it connects (Connected Agents > 0 on the ACC WebSocket Endpoint stats). 4. Once confirmed working, roll out the acc.yml change to the rest of your agents. As long as the F5 VIP cert has a wildcard, it should already cover the hyphenated hostname - you do NOT need a new certificate. ***This issue is being addressed in PRB2024847