On-Call Rotation stopped sending more than one notificationDescription<!-- div.margin{ padding: 10px 40px 40px 30px; } table.tocTable{ border: 1px solid; border-color:#E0E0E0; background-color: rgb(245, 245, 245); padding-top: .6em; padding-bottom: .6em; padding-left: .9em; padding-right: .6em; } table.noteTable{ border:1px solid; border-color:#E0E0E0; background-color: rgb(245, 245, 245); width: 100%; border-spacing:2; } table.internaltable { white-space:nowrap; text-align:left; border-width: 1px; border-collapse: collapse; font-size:14px; width: 85%; } table.internaltable th { border-width: 1px; padding: 5px; border-style: solid; border-color: rgb(245, 245, 245); background-color: rgb(245, 245, 245); } table.internaltable td { border-width: 1px; padding: 5px; border-style: solid; border-color: #E0E0E0; color: #000000; } .title { color: #D1232B; font-weight:normal; font-size:28px; } h1{ color: #D1232B; font-weight:normal; font-size:21px; margin-bottom:-5px } h2{ color: #646464; font-weight:bold; font-size:18px; } h3{ color: #000000; font-weight:BOLD; font-size:16px; text-decoration:underline; } h4{ color: #646464; font-weight:BOLD; font-size:15px; text-decoration:; } h5{ color: #000000; font-weight:BOLD; font-size:13px; text-decoration:; } h6{ color: #000000; font-weight:BOLD; font-size:14px; text-decoration:; } ul{ list-style: disc outside none; margin-left: 0; } li { padding-left: 1em; } --> Symptoms After creating Incident, the escalation document sends the first notification, then never sends another notification. On-Call Rotation stopped sending more than one notification Release London Patch 1 Hot Fix 2 Cause Reviewing node logs in Debug we can see after BR: 'mk Oncall Notify', BR: mkOnCallEscalation" triggered. These are custom Business Rules. Reviewed Table "cmn_rota_escalation" and found escalation document. XML document shows following: <?xml version="1.0" encoding="UTF-8"?> <escalations current_rota="778a331b957e61005d67c5f9f488ca38" group="81a6abf16f57590057d79b9eae3ee47b" primary_user_id="8edaef1a6fd01a040b3f687f8e3ee433" primary_user_name="Test User" processed_rota_list=""> <notifications> <sent delay="0" member="5ff93f346fdd924457d79b9eae3ee4c2" notified="2018-12-06 02:47:36" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086056540" run_at_user_time="2018-12-06 02:47:36" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> <pending delay="30" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086086540" run_at_user_time="2018-12-06 02:48:06" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> <pending delay="60" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086116540" run_at_user_time="2018-12-06 02:48:36" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> <pending delay="90" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086146540" run_at_user_time="2018-12-06 02:49:06" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> </notifications> </escalations> The escalation document shows first notification sent after delay=0 successfullyHowever after delay="30", delay="60", delay="90" all showing Pending. Resolution Upgrade to LP2 as it includes PRB1296458 as fix. However, in this case customer was not ready to upgrade to LP2 hence provided following workaround. NOTE: Notified customer that updates against 3 script includes need to be reverted back on upgrade so that they will continue to get fixes and enhancements. Workaround Provided by Dev. OnCallRotation: =============== _checkForMoreRotas: function() { this._escalationList.addRotaToList(this._rotaSysId); var time = new GlideDateTime(); this._firstDelay = this._getFirstDelay(time); if (this._rotaSysId) { time.addSeconds(this._firstDelay); this._getRotaNotificationsByTime(time, true); } var moreRotas = !this.isEmpty(); if (this._log.atLevel(GSLog.DEBUG)) this._log.debug("[_checkForMoreRotas] moreRotas: " + moreRotas); return moreRotas; }, /** * Get the set of notifications for the current rota based on the rota's escalation type and rotation members * gdt [GlideDateTime] the time point to get notifications for * nullifyOverrideRoster [boolean] */ _getRotaNotificationsByTime: function(gdt, nullifyOverrideRoster) { var gr = new GlideRecord('cmn_rota_roster'); gr.addQuery("rota", this._rotaSysId); gr.addActiveQuery(); gr.query(); this._getRotaNotifications(gr, gdt, nullifyOverrideRoster); }, ================ OnCallEscalation ================ /** * Load up the escalation information from the xml string */ fromXml: function(escalationXmlStr) { this.clear(); if (!escalationXmlStr) return; var doc = GlideXMLUtil.parse(escalationXmlStr); if (!doc) return; var root = doc.getDocumentElement(); this.groupSysId = root.getAttribute("group"); this.rotaSysId = root.getAttribute("current_rota") + ""; this.primaryUserSysId = root.getAttribute("primary_user_id"); this.primaryUserName = root.getAttribute("primary_user_name"); this.rotaList = []; var rotaList = root.getAttribute("processed_rota_list").split(","); for(var k = 0; k < rotaList.length; k++) { this.rotaList.push(rotaList[k]); } var pending = GlideXMLUtil.selectNodes(root, "/escalations/notifications/" + OnCallEscalationEntrySNC.PENDING); for (var i = 0, length = pending.getLength(); i < length; i++) { var pendingEntry = new OnCallEscalationEntry(); pendingEntry.fromXml(pending.item(i)); var key = this.createKey(pendingEntry.getDelay()); this.entries[key] = pendingEntry; this.entryKeys.push(key); } var sent = GlideXMLUtil.selectNodes(root, "/escalations/notifications/" + OnCallEscalationEntrySNC.SENT); var sentLength = sent.getLength(); for (var j = 0; j < sentLength; j++) { var sentEntry = new OnCallEscalationEntry(); sentEntry.fromXml(sent.item(j)); this.notifiedList.push(sentEntry); if (j === 0) this.firstNotifiedGdt = new GlideDateTime(sentEntry.getNotifiedTime()); } // Start the iterator at the beginning of the list this.gotoTop(); }, ===================== OnCallEscalationEntry ===================== toXml: function(doc, parentElement) { parentElement.appendChild(this._createElement(doc)); }, Additional Information As a reminder, updates against 3 script includes need to be reverted back on upgrade so that customer will continue to get fixes and enhancements.