mid_server.down events running script action 'Fail over MID server' where there is a large ECC Queue backlog can block instance system event queue processing - Known Error

mid_server.down events running script action 'Fail over MID server' where there is a large ECC Queue backlog can block instance system event queue processing Description When a MID Server goes down for any reason, a mid_server.down event is fired, which triggers the script action 'Fail over MID server'. This is run by the "events process 0" scheduled job, as part of processing a batch of system events. If, for whatever reason, the ECC Queue table is huge, and has many ready jobs for the mid server that has gone down, then this can hold up events processing for a long time, and the other events in that batch will not be processed until the Fail over MID server script action has finished. The events process may need to be killed by support before those can run.

The purpose of this job is to either update the ECC output record to have it picked up by another mid server in the same failover cluster, or more recently to use MID Selection based on the IP range/Applications/Capabilities to pick a different MID Server. For a huge number of records, that can be a long job.

Note: This problem ticket is for the impact on the instance and its events processing in general, and not for the MID Server going down or having a backlog in the first place.

Steps to Reproduce

Install 2 MID Servers Build up an output/ready backlog in the ecc_queue for one of the MID Servers, of say a million records Turn that mid server off. This will result in a long running events process job

While that is running, turn the same mid server back on and off again. If the instance has multiple events process jobs, it is likely to cause another job, updating the same records, both running at the same time, which will compound the database and instance performance, as there are no checks or tracking for what's already running, or what's been happening with the mid server since the original down event.

All Active Transactions for an app node may show one or more of these running as system, doing queries on ecc_queue000x and ecc_agent_cluster_member_m2m: /v_transaction_list.do user, url, age, state, DB time, sql statements system, events process 0, 19 Hours 53 Minutes, Business Rules, 10 Hours 30 Minutes, 46,106,034, 0 Seconds 0 javascript:gs.processDelegatedEvents(); system, events process 0, 1 Day 12 Hours 15 Minutes, Database, 9 Hours 10 Minutes, 88,240,665, 0 Seconds 1 SELECT ecc_agent_cluster_member_m2m 0.`cl... system, events process 0, 1 Day 18 Hours 18 Minutes, Database, 22 Hours 12 Minutes, 103,247,573, 0 Seconds 48 ( SELECT ecc_queue 00000.`agent`, ecc_queu...

"events process 1" and other names may be seens for the same thing.

Stack trace of the events process 0 threads will look like this if looping through ecc_queue records:

main,glide.scheduler.worker.x,x java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) ... com.glide.db.meta.TableIterator.next(TableIterator.java:55) com.glide.script.GlideRecordITable.next(GlideRecordITable.java:459) com.glide.script. GlideRecord.next (GlideRecord.java:6480) com.glide.script.GlideRecord.nextFromJS(GlideRecord.java:5931) com.glide.script.GlideRecord.js__next(GlideRecord.java:5926) com.glide.script.GlideRecord.jsFunction_next(GlideRecord.java:5906) sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138) org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670) org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582) org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:85) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_199743._c_failOver_1( sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script:21 ) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_199743.call(sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:74) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_199743._c_script_0(sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script:7) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_199743.call(sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script) ... or this if working out the available clusters/mid servers to use instead:

main,glide.scheduler.worker.x,x org.mozilla.javascript.ScriptableObject.get(ScriptableObject.java:466) org.mozilla.javascript.IdScriptableObject.get(IdScriptableObject.java:337) com.glide.script.ARhinoScope.get(ARhinoScope.java:68) com.glide.script.RhinoScope.get(RhinoScope.java:52) org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:2315) org.mozilla.javascript.ScriptableObject.getClassPrototype(ScriptableObject.java:2205) org.mozilla.javascript.TopLevel.getBuiltinPrototype(TopLevel.java:137) org.mozilla.javascript.ScriptableObject.getObjectPrototype(ScriptableObject.java:2168) org.mozilla.javascript.BaseFunction.getClassPrototype(BaseFunction.java:356) org.mozilla.javascript.BaseFunction.getObjectMeta(BaseFunction.java:23) org.mozilla.javascript.Context.getObjectMetaData(Context.java:2983) org.mozilla.javascript.ScriptRuntime.getMeta(ScriptRuntime.java:2553) org.mozilla.javascript.ScriptRuntime.getMetadata(ScriptRuntime.java:2567) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2607) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.ScriptableObject.getDefaultValue(ScriptableObject.java:965) org.mozilla.javascript.ScriptableObject.getDefaultValue(ScriptableObject.java:905) com.glide.script.GlideElement.getDefaultValue(GlideElement.java:1754) org.mozilla.javascript.ScriptRuntime.add(ScriptRuntime.java:2854) org.mozilla.javascript.gen.sys_script_include_f6c69a020a0006bc36db905d8d02dfc2_script_543._c_anonymous_10( sys_script_include.f6c69a020a0006bc36db905d8d02dfc2 .script:181) org.mozilla.javascript.gen.sys_script_include_f6c69a020a0006bc36db905d8d02dfc2_script_543.call(sys_script_include.f6c69a020a0006bc36db905d8d02dfc2.script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:85) org.mozilla.javascript.gen.sys_script_include_f6c69a020a0006bc36db905d8d02dfc2_script_543._c_anonymous_1(sys_script_include.f6c69a020a0006bc36db905d8d02dfc2.script:37) org.mozilla.javascript.gen.sys_script_include_f6c69a020a0006bc36db905d8d02dfc2_script_543.call(sys_script_include.f6c69a020a0006bc36db905d8d02dfc2.script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2549) org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:300) org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:102) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) org.mozilla.javascript.gen.sys_script_include_d22e7bdbc0a8016500a18e024bfc9aa3_script_74._c_anonymous_2(sys_script_include.d22e7bdbc0a8016500a18e024bfc9aa3.script:4) org.mozilla.javascript.gen.sys_script_include_d22e7bdbc0a8016500a18e024bfc9aa3_script_74.call(sys_script_include.d22e7bdbc0a8016500a18e024bfc9aa3.script) org.mozilla.javascript.BaseFunction.construct(BaseFunction.java:372) org.mozilla.javascript.ScriptRuntime.newObject(ScriptRuntime.java:2449) org.mozilla.javascript.ScriptRuntime.newObjectEx(ScriptRuntime.java:2464) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_191645._c_failOver_1( sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script:119 ) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_191645.call(sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:74) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_191645._c_script_0(sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script:7) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_191645.call(sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script) org.mozilla.javascript.gen.sysevent_script_action_f6c24d230a0006bc394931345fba7a8a_script_191645.exec(sysevent_script_action.f6c24d230a0006bc394931345fba7a8a.script) ... Workaround This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.