Event Rule changes can cause event_rule.update_events to block system events processing, causing a instance-wide performance issue - Known Error

Event Rule changes can cause event_rule.update_events to block system events processing, causing a instance-wide performance issue Description When the Event Management feature's Event Rules [em_match_rule] are inserted/updated or deleted, the "update empty events"/"Handle deleted event rule" business rule fires a "event_rule.update_events"/"event_rule.update_events_after_delete" system event in the default events queue, to re-check any em_event records not yet covered by any previously existing event rule. Script action "Update event rules for empty events"/"Update rules after deletion of ER" runs for the event. This event is executed by the default Events queue scheduled job "Events process 0", which will also be processing batches of system events from the platform and other features too.

When an instance has a large number of em_event records matching the new rule criteria, which is likely, then this will be a long running transaction.

Note: Event Management feature's input Events [em_event] and the Platform's System Events [sys_event] are very different things, not to be confused with each other.

Steps to Reproduce

In an instance with Event Management installed Insert/Update or Delete an Event Rule, when there will be a considerable number of existing em_event record that it will now match with. Expected behaviour: The default event queue should not be used for this kind of job. If the job runtime cannot be optimised, or batched, or implemented some other way, then normally this type of event would be registered for a event management-specific queue, or even an event-specific queue, and a dedicated scheduled job set up for that queue.

Actual behaviour: In customer instances, these events have been seen to take over 17 minutes to process, blocking the "Events process 0" job of the app node, delaying processing of other important events that should be being processed in near real-time, or many different problems in many apps will be caused. Delayed event processing is a serious issue, that will cause monitoring alerts and proactive P1/2 support cases for the instance.

/threads.do for the default Events queue scheduled job "Events process 0", will show it stuck in function EventRuleHandler checkAndUpdateEmptyEvents for "Update event rules for empty events", and checkAndUpdateEmptyEventsPerRule for "Update rules after deletion of ER"

main,glide.scheduler.worker.0,x ... com.glideapp.itom.snac.rules.MatchRule.match(MatchRule.java:245) com.glideapp.itom.snac.rules.MatchRule.matches(MatchRule.java:237) com.glideapp.itom.snac.rules.EventRuleEngine.getFilteredMatchedRules(EventRuleEngine.java:615) com.glideapp.itom.snac.rules.EventRuleEngine.executeMatchMultipleRules(EventRuleEngine.java:516) com.snc.sw.eventrule.handler.EventRuleHandler.updateEvents(EventRuleHandler.java:234) com.snc.sw.eventrule.handler.EventRuleHandler.updateLimitedEvents(EventRuleHandler.java:198) com.snc.sw.eventrule.handler.EventRuleHandler.checkAndUpdateEmptyEvents(EventRuleHandler.java:187) com.snc.sw.eventrule.EventRuleManager.jsFunction_checkAndUpdateEmptyEventsPerRule(EventRuleManager.java:54) java.base@17.0.12/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) java.base@17.0.12/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) java.base@17.0.12/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.base@17.0.12/java.lang.reflect.Method.invoke(Method.java:569) org.mozilla.javascript.MemberBox.invoke(MemberBox.java:151) org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:717) org.mozilla.javascript.FunctionObject.call(FunctionObject.java:654) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2973) org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:55) org.mozilla.javascript.gen.sysevent_script_action_bd02337393413200a346f179077ffbda_script_66523._c_script_0(sysevent_script_action.bd02337393413200a346f179077ffbda.script:2) org.mozilla.javascript.gen.sysevent_script_action_bd02337393413200a346f179077ffbda_script_66523.call(sysevent_script_action.bd02337393413200a346f179077ffbda.script) org.mozilla.javascript.gen.sysevent_script_action_bd02337393413200a346f179077ffbda_script_66523.exec(sysevent_script_action.bd02337393413200a346f179077ffbda.script) ... com.glide.script.GlideSystem.js_processDelegatedEvents(GlideSystem.java:793) ... com.glide.schedule_v2.SchedulerWorkerThread.run(SchedulerWorkerThread.java:172) Workaround This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available.

Simple workaround :

1. go to the event rules table

2. go to business rules

3. disable Handle deleted event rule and update empty events rules

An advanced workaround to avoid these events blocking the default events queue processing would be to create a custom queue, and custom events processor job, along the lines of KB0676909 Create a Dedicated Event Processor

Note: This change will be considered a customisation by the Upgrade engine, and so these changes will need deleting/reverting after you upgrade the instance to the fixed version to avoid problems in future.

Create the Scheduled job first, so that it is already there when you start sending events to it later. navigate to System Scheduler > Scheduled Jobs module, then click the New button. Enter the following values for the fields and then Click Submit. (Swap out QUEUENAME for the queue name set in step 4) Name: event_rule events process System ID: -- None -- Job Context: fcScriptName=javascript\:GlideEventManager(' event_rule ').process(); Trigger type: Interval Repeat: 30 seconds (Leave all other fields as default. Ensure that the Job ID field defaults to RunScriptJob) Then update the pair of Event Registry records to use the new queue instead Navigate to System Policy > Registry module For each of these: update empty events https:// .service-now.com/sysevent_register.do?sys_id=95a1bf3393413200a346f179077ffb71 Handle deleted event rule https:// .service-now.com/sysevent_register.do?sys_id=cb3385c493513200a346f179077ffb0c Update the Queue field Queue: event_rule Save Confirm it has worked, by making a change to an Event Rule Navigate to System Logs > Events module filter down for event name starts "event_rule.update_events" Check that the newly created events are being processed Check that the newly created events are going into the correct queue Related Problem: PRB1838597