Reconciliation and other parallel job threads created on one node - Known Error

Reconciliation and other parallel job threads created on one node Description Recon job workers threads gets creates on only one node which is causing spikes and issues in customer's instance

Steps to Reproduce

Run recon and observe that most of the times, the worker nodes gets created on one node

Below logs are from marriot customer: marriott221 glide.scheduler.worker.6 Import Set Transformer 2022-10-29 20:03:54 1 day, 12:53:30 marriott212 glide.scheduler.worker.4 SAM - Software License Reconciliation Wo 2022-10-31 05:00:43 3:56:47 marriott212 glide.scheduler.worker.5 SAM - Software License Reconciliation Wo 2022-10-31 05:00:43 3:56:47 marriott212 glide.scheduler.worker.2 SAM - Software License Reconciliation Wo 2022-10-31 05:00:43 3:56:47 marriott212 glide.scheduler.worker.1 SAM - Software License Reconciliation Wo 2022-10-31 05:00:43 3:56:47 marriott211 glide.scheduler.worker.2 Flow Engine Event Handler 2022-10-31 06:07:11 2:50:13 marriott212 glide.scheduler.worker.3 Service Mapping - Traffic Process to Pro 2022-10-31 06:43:55 2:13:35 marriott201 glide.scheduler.worker.7 Hotel Open Incident Daily 2022-10-31 08:00:08 0:57:16 marriott217 glide.scheduler.worker.6 Flow Engine Event Handler 2022-10-31 08:07:11 0:50:13 marriott212 glide.scheduler.worker.0 Email Reader 2022-10-31 08:34:10 0:23:20 marriott212 glide.scheduler.worker.6 SMS Sender 2022-10-31 08:45:35 0:11:55 marriott201 glide.scheduler.worker.6 ciforensics events process 2022-10-31 08:49:30 0:07:54 marriott202 glide.scheduler.worker.2 Autoclose Incidents 2022-10-31 08:55:00 0:02:24 marriott218 glide.scheduler.worker.7 System Trigger2022-10-31 12:55:38 2022-10-31 08:55:38 0:01:46 marriott212 glide.scheduler.worker.7 text index events process 2022-10-31 08:57:14 0:00:16

Workaround In the ReconciliationEngine script include, find the method startReconciliationWorkers() and add a line "gs.sleep(1000)" to delay the start of each worker at the end of the for loop. See the attached photo. For other parallel jobs (deduplication, software model to discovery model matching, lifecycle), make the same change to their corresponding script includes and 'start worker' method as desired.