[CPG] Exception handling is not proper when provisioning takes longer time. Stack remains Active and Request is stuck in processing - Known Error

[CPG] Exception handling is not proper when provisioning takes longer time. Stack remains Active and Request is stuck in processing Description When VM provisioning requests are made, the RITM requests stuck in 'processing' state for ever, though in midserver logs we see that the response has been received successfully.

This happens when the VM provisioning takes more than 30 mts but due to the mid property : mid.property.probe.api.timeout.minutes set to 30 mts, the request times out. Even if correct response is received later, this does not matter anymore as the request has already timed out.

This is an example log of the occurence of this issue in the midserver logs :

2023-11-02 12:24:38 INFO (Worker-Standard:APIProxyProbe-88e9b0cd33563d10067043bfcd5c7b86) [APIProxyProbe:289] About to start route... .... Nothing happened, no response yet from Terraform up until 12:56 below ........ 2023-11-02 12:55:23 INFO (Worker-Standard:APIProxyProbe-88e9b0cd33563d10067043bfcd5c7b86) [APIProxyProbe:307] Route 'system-94aea4ce-f05c-4eae-b1f5-7937b49b0346' removed successfully: 'true'. 2023-11-02 12:55:23 ERROR (Worker-Standard:APIProxyProbe-88e9b0cd33563d10067043bfcd5c7b86) [APIProxyProbe:384] Could not initialize APIProxyProbe. Details - null 2023-11-02 12:55:23 ERROR (Worker-Standard:APIProxyProbe-88e9b0cd33563d10067043bfcd5c7b86) [AProbe:315] com.snc.automation_common.integration.exceptions.UnknownException: Could not initialize APIProxyProbe at com.snc.cmp.mid.probe.APIProxyProbe.probe(APIProxyProbe.java:385) at com.service_now.mid.probe.AProbe.process(AProbe.java:154) .... the probe already completed itself after 30 minutes ...... with empty result ..... Then, the actual message from Terraform arrives: 2023-11-02 12:56:47 INFO (Camel (camel-28) thread #53 - ProducerTemplate) [MIDSystem:35] *** Script: *** INFO: (TerraformExecutor): Output from cloud script:{"status":"success","output":{"terraform":{"init":"Initializing the backend... Initializing provider plugins... - Reusing previous version of hashicorp/vsphere from the dependency lock file - Using previously-installed hashicorp/vsphere v2.4.3 Terraform has been successfully initialized! You may now begin working with Terraform. Try running \"terraform plan\" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.","workspace.new":"Created and switched to workspace...

Steps to Reproduce

1. Login to instance : 2. Create ARM/ CFT Catalog Go to Mid server script include -> Azure Deployment or AWs Deployment comment the below line in _waitForDeployment method var deployInfo = this.getResource(deploymentId, false); or Add a Sleep for 30 mins.

4. provision catalog, waiut for 30 mins, you will get an exception in the mid log. 5. Stack stays active/stuck in processing state forever. Response mapping also in progress state. ITs not shown in Error state.

Ideally it should show in Error state.

Workaround There is a timeout on the mid-server level, governed by the "mid.property.probe.api.timeout.minutes" mid property with a default value of 30 mins. Increasing this value to above 30mts should temporarily fix the issue.