Global IP Exclusion causes slowness in Discovery Schedules with a large number of Discovery IP Ranges - Slow Starting, and slow/timeout Shazzam sensors causing stuck Discovery Status - Known Error

Global IP Exclusion causes slowness in Discovery Schedules with a large number of Discovery IP Ranges - Slow Starting, and slow/timeout Shazzam sensors causing stuck Discovery Status Description A Discovery Status can be in Starting state for a long time before the first Shazzam ecc_queue output is created and it becomes Active state, when there are lots of Global IP Exclusion records, and a large number of IP ranges on the Discovery Schedule.

For example, 4 hours to become Active and start seeing probes, with 23k Global IP Exclusion and 1400 IP ranges. Also 30 minutes with 50k Global IP Exclusion records, and 150 IP ranges.

This will also delay Shazzam sensor processing times, while those are also creating new Shazzam probes for the next batch of IPs. That can time out at 20 minutes due to the quota rule, meaning the sensor is not counted as completed, and the Discovery Status will never get set to Completed due to the started/completed count mismatch.

Steps to Reproduce

Create a Discovery Schedule of type CI/Networks/IP Addresses Add 2000 discovery_range_items of type IP Networks all with netmask values in the range of 29-32 Add 20000 records to ip_exclusion table that are not related that are different from the IPs in discovery schedule created in steps 1, 2 Run Discovery in debug mode, notice that we are calling the getRange method 2000 times, once for each range being added to the ranges list during Shazzam Make note of time taken for Discovery status to transition from Starting to Active. Also note subsequent Shazzam Sensor processing times, and look our for cancelled transactions due to the quota rule, In that situation, the undercounting of completed sensors will cause a stuck Discovery Status that never gets set to Completed. Where Quota rule kills the sensor transaction, the Shazzam ecc_queue input will be State=Error, Error string: Sensor error: Transaction cancelled: maximum execution time exceeded

You will also see these cancellations reported in syslog_cancellation.

Appnode localhost logs will show something like this, where this one also includes some memory watcher warnings:

2023-06-11 03:28:40 (403) worker.0 worker.0 txid=bc284b2efef0 Discovery: Processing sensor: Shazzam (See Payload) b0280b62871f6dd056f72f4acebb35c2 2023-06-11 03:28:41 (028) worker.0 worker.0 txid=bc284b2efef0 [0:00:00.016] Compacting large row block (file.write: ip_exclusion 10000 rows 160000 saveSize) 2023-06-11 03:28:41 (063) worker.0 worker.0 txid=bc284b2efef0 [0:00:00.004] Expanding large row block (file.read: ip_exclusion, 10000 rows, 160000 dataSize) 2023-06-11 03:46:34 (365) glide.memory.watcher SYSTEM URL= JOB: ASYNC: Discovery - Sensors ssh,wmi (See Payload) b0280b62871f6dd056f72f4acebb35c, THREAD= glide.scheduler.worker.0, FG= false, TYPE= 4, STATE= 4, USER= system, TIME= 1,074,036, MEM= 0, ATTRIBUTES= {ecc_queue_sys_id=b0280b62871f6dd056f72f4acebb35c2, discovery_status_sys_id=7912cb26875f6554be0cfd17cebb352e} 2023-06-11 03:48:41 (270) glide.quota.manager SYSTEM WARNING *** WARNING *** Transaction: Cancelling transaction ASYNC: Discovery - Sensors ssh,wmi (See Payload) b0280b62871f6dd056f72f4acebb35c - system (maximum execution time exceeded): Thread glide.scheduler.worker.0 (system, glide.scheduler.worker.0), after 1200941ms 2023-06-11 03:48:41 (271) worker.0 worker.0 txid=bc284b2efef0 WARNING *** WARNING *** Long Transaction started at 06/11/23 03:28:40.329, Memory at start was 1,357, Memory is 1,311, SQL count is 96,761, BR count is now 65. 2023-06-11 03:48:41 (271) worker.0 worker.0 txid=bc284b2efef0 WARNING *** WARNING *** Transaction cancelled: maximum execution time exceeded 2023-06-11 03:48:41 (276) worker.0 worker.0 txid=bc284b2efef0 SEVERE *** ERROR *** *** Script: SensorProcessor failed for ECC queue record undefined - com.glide.sys.TransactionCancelledException: Transaction cancelled: maximum execution time exceeded 2023-06-11 03:48:41 (289) worker.0 worker.0 txid=bc284b2efef0 Completed: ASYNC: Discovery - Sensors ssh,wmi (See Payload) b0280b62871f6dd056f72f4acebb35c.3c280b62871f6dd056f72f4acebb35c2 in 0:20:00.943, next occurrence is null, sys_created_on: 2023-06-11 10:28:40, trigger_type: 0 (Run Once), priority: 105, run_time: , repeat: , previous next_action time: 2023-06-11 10:28:40.000 UTC, current next_action time: null, max_drift: null, queued on: 2023-06-11 10:28:40.327 UTC, execution time millis: 1200943, actual start time: 2023-06-11 10:28:40.334 UTC, actual end time: 2023-06-11 10:48:41.277 UTC Workaround This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available.