MID Servers repeatedly Upgrade and Downgrade between current and previous instance version - Support and Troubleshooting

MID Servers repeatedly Upgrade and Downgrade between current and previous instance version Issue  Symptoms After an instance upgrade, patch or hotfix, the MID Servers should upgrade themselves to match. MID Servers may upgrade to the new version, but then over the next hours or days downgrade again. They may then repeat that cycle.

Release Any

Cause Most instances are made up of more than one application node. The load balancer will share the incoming requests from MID Servers among them, making it random which node a MID Server is currently connected to.

When an instance upgrades, each of those nodes is upgraded and restarted in turn to avoid an outage, but if something goes wrong then a node may remain running the previous version , and report that previous version to the MID Server when the MID Server checks what version it should be running. PRB1299098 and PRB1278372 have been known to cause this prior to the Madrid release, but once upgraded to Madrid you won't see those again.

The Stats page (/stats.do) is specific to the node, and the clue is that you may see the previous version still reported on that page. It is not usually possible to select which node you log into so having several of your users check stats.do may be required before you see all the nodes.

The "Upgrade Monitor" page "Node Upgrades" section, will also show if any nodes are not upgraded.

Example: These are the bits from the top of the /stats.do pages from a normal 2 node instance, which was upgraded from "Kingston Patch 6" to "Kingston Patch 6 Hotfix 5" on Sep 01.

Note how this node 001 reports the correct version, and indicates that it was restarted as part of the upgrade process on Sep 01. A MID Server connecting to this node will Upgrade.

Statistics for: Demo Server @ appXXXXXX.iadXXX.service-now.com:16054 at: Wed Sep 05 06:17:09 PDT 2018 (Refresh) Connected to cluster node: appXXXXXX.iadXXX.service-now.com: instancename001 Build name: Kingston Build date: 07-31-2018_0903 Build tag: glide- kingston-10-17-2017__patch6-hotfix5 -07-28-2018 I nstance name: instancename Instance ID: 123456510a0a3c560094215d720219d5 Node ID: 3d27d8123456c66530025f6c009930cb IP address: 10.XX.XX.XX MID buildstamp: kingston-10-17-2017__patch6-hotfix5-07-28-2018_07-31-2018_0903

Servlet Statistics Started: Sat Sep 01 13:24:13 PDT 2018 This is node 002, and is still reporting the previous version. It was not restarted on the day of the upgrade. The MID Server will not upgrade after the instance upgrade, or downgrade again, if it happens to connect to this node.

Statistics for: Demo Server @ appXXXXXX.iadXXX.service-now.com:16047 at: Wed Sep 05 06:17:10 PDT 2018 (Refresh) Connected to cluster node: appXXXXXX.iadXXX.service-now.com: instancename002 Build name: Kingston Build date: 05-24-2018_1317 Build tag: glide- kingston-10-17-2017__patch6 -05-16-2018 Instance name: instancename Instance ID: 123456510a0a3c560094215d720219d5 Node ID: 3d27d8123456c66530025f6c009930cb IP address: 10.XX.XX.XX MID buildstamp: kingston-10-17-2017__patch6-05-16-2018_05-24-2018_1317

Servlet Statistics Started: Fri Aug 17 10:39:52 PDT 2018 Resolution This is not something customers should try to resolve themselves. Please open a Case with Customer Support to have the Upgrade issue fixed.

Once all nodes are reporting the current version, then the MID Servers will upgrade to that version too, and stay that way.