MID Server upgrade fails leaving MID Server Down, due to Cisco Advanced Malware Protection (AMP) for Endpoints preventing MID Server upgrade process replacing the Wrapper executable file.
The Upgrade starts, the MID Server launches the dist upgrade process and shuts itself down. The Dist-upgrade process failed with FileNotFoundException and stops due to the wrapper-windows-x86-64.exe file being locked/blocked by Cisco AMP.
NOTE: This PRB is specific to MID Server outages caused by Cisco AMP during upgrades, but this symptom is not always caused by Cisco AMP. The same symptom has also been reported on server not running Cisco AMP.
Before applying the work-around we need to make sure that the root cause is Cisco AMP. To do this we need to verify the wrapper.log and also make sure that Cisco AMP is running.
Verifying the error message in wrapper.log
The MID Server wrapper.log will show this at the end (assuming no manual attempt was made to start it since):
May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain run SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied) com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied) at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:840) at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:313) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream.(FileOutputStream.java:162) at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1142) at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1446) at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1444) at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1388) at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1317) at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:837) ... 2 more May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain appendMidLogs INFO: Flushing logs << UPGRADE LOG END >>
Verifying Cisco AMP is running
Open "Task Manager" and make sure that CiscoAMP is running
To resolve the issue, you need to add an exclusion set including the following exclusions to the policy applied for the MID Server host machine on Cisco AMP Console:
To add the exclusions