SG-GCP rate limiting cross-contamination between multiple ServiceNow instances (429 -> -1 timeout)<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Applies To ServiceNow instances integrated with Google Cloud Platform (GCP) Service Graph Connector AKA SG-GCP / Cloud Asset Inventory API, where more than one ServiceNow instance connects to the same GCP organization. Summary This article documents an observed failure pattern where a GCP API rate limit (HTTP 429) triggered by one ServiceNow instance can cause a completely different ServiceNow instance – connected to the same GCP org – to experience what appears on our side as a silent socket timeout (response_status = -1), rather than a visible 429 response. NOTE: This should not discount the chance of just requesting to much data at once, this can be reduced via the respective batch size property. This article covers the other possibility in the scenario of a -1 response code. In terms of the property, the respective data source will have one named with in the SgGcpConstants sys_script_Include, the name will align with the data source. Background It is common for a single customer to run multiple ServiceNow instances (for example, DEV, TEST, and PROD, or separate business-unit instances) that are all configured to pull asset data from the same underlying GCP organization/"tenancy" via the Cloud Asset Inventory API (cloudasset.googleapis.com). This is the API SG-GCP pulls many different types of CIs from. GCP applies rate limiting policies at the API/quota level, scoped to the GCP organization being queried, some enterprise orgs can control by project – not to the individual calling client. This means the rate limit budget is shared across every consumer hitting that API for that org, regardless of which ServiceNow instance is making the call. The Logic – How a 429 on Instance A Becomes a -1 on Instance B 1. Instance A and Instance B both have active scheduled jobs (SG-GCP data source loads via scheduled jobs) querying the same GCP org at or near the same time. 2. Instance A's call volume (or likely combined volume from both instances) trips GCP's rate limiter for that org/API policy. GCP responds to Instance A with an HTTP 429 (Too Many Requests). 3. Instance B has an API call already in flight against that same rate-limited policy. Because the quota bucket is shared, GCP does not necessarily send Instance B a 429 back. Instead, GCP may simply withhold sending a response body for Instance B's already-accepted request. This looks like a call that just ends with a -1 response code. 4. From ServiceNow's perspective on Instance B: the TCP handshake completed successfully and GCP accepted the HTTP request (the connection was established), but no data is ever sent back after that point. 5. Our HTTP client on Instance B waits for a response until its configured socket timeout is reached. Once that timeout expires, we forcibly close the socket on our end. 6. Because there was never a proper HTTP response (no status line, no body), this is logged on our side as: – response_status = -1 – response_length = -1 – response_time = (full timeout duration, e.g. ~60000ms as default) – HTTPClient WARNING: Socket error: Socket closed 7. This -1 is functionally different from a normal error response. It does not mean GCP rejected the request outright (that would show as a 4xx/5xx status). It means GCP accepted the connection but never completed the response – consistent with the instance being caught behind another consumer's rate limit event on the same quota. Key Takeaway A -1 socket-closed result on one ServiceNow instance's GCP integration log does not always indicate a problem local to that instance or that specific job run. If the customer runs multiple ServiceNow instances against the same GCP organization, the root cause may be a rate limit event triggered by a sibling instance sharing the same GCP quota policy. Diagnostic Guidance When troubleshooting a -1 / socket-closed error on a GCP data source load: • Customer needs to review if they have more than one ServiceNow instance integrated with the same GCP org. • If yes, check whether any of those instances logged an explicit 429 response around the same timestamp as the -1 event on the affected instance. • Correlate timestamps across instances where possible – a 429 on Instance A occurring just before or during the -1 window on Instance B supports this pattern. • Do not treat the -1 in isolation as a connectivity or firewall issue without first ruling out shared-tenancy rate limit contention. Mitigation Options • Stagger scheduled job run times across instances connected to the same GCP org, so they are less likely to overlap and compete for the same rate limit window. • Engage the customer's GCP admin team to review and, if possible, increase the quota/rate limit ceiling for the shared org if multiple ServiceNow instances are a permanent architecture decision. Related Notes • This is distinct from a standard connection/authentication failure (expired credentials, bad endpoint) – those return a normal HTTP error status, not a -1. • This is also distinct from a large-payload OOM condition – those manifest as memory warnings, not socket timeouts.