MID Server upgrade fails leaving MID Server Down, due to Cisco AMP antivirus preventing the upgrade service deleting the Wrapper executable - Known Error

MID Server upgrade fails leaving MID Server Down, due to Cisco AMP antivirus preventing the upgrade service deleting the Wrapper executable Description MID Server upgrade fails leaving MID Server Down, due to Cisco Advanced Malware Protection (AMP) for Endpoints preventing MID Server upgrade process replacing the Wrapper executable file.

The Upgrade starts, the MID Server launches the dist upgrade process and shuts itself down. The Dist-upgrade process failed with FileNotFoundException and stops due to the wrapper-windows-x86-64.exe file being locked/blocked by Cisco AMP.

NOTE: This PRB is specific to MID Server outages caused by Cisco AMP during upgrades, but this symptom is not always caused by Cisco AMP. The same symptom has also been reported on server not running Cisco AMP.

Before applying the work-around we need to make sure that the root cause is Cisco AMP. To do this we need to verify the wrapper.log and also make sure that Cisco AMP is running.

Verifying the error message in wrapper.log

The MID Server wrapper.log will show this at the end (assuming no manual attempt was made to start it since):

May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain run SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied) com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied) at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:840) at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:313) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\ wrapper-windows-x86-64.exe (Access is denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream.(FileOutputStream.java:162) at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1142) at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1446) at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1444) at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1388) at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1317) at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:837) ... 2 more May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain appendMidLogs INFO: Flushing logs >

Verifying Cisco AMP is running

Open "Task Manager" and make sure that CiscoAMP is running

Steps to Reproduce

Install a MID Server on a Windows host running Cisco Advanced Malware Protection (AMP) Cause the MID Server to upgrade Some upgrades will fail to upgrade at the point that the old agent\bin\wrapper-windows-x86-64.exe is deleted Workaround To resolve the issue, you need to add an exclusion set including the following exclusions to the policy applied for the MID Server host machine on Cisco AMP Console:

File Scan for wrapper-windows-x86-64.exe under agent\bin folder (with Apply to child processes marked) File Scan for java.exe under agent\jre\bin folder (with Apply to child processes marked) Wildcard for the MID Server folder To add the exclusions

Make sure you have proper privilege to add an exclusion to Cisco AMP On the Cisco AMP console Create a new Exclusion set Add a new "File Scan" exclusion to the exclusion set to exclude the wrapper-windows-x86-64.exe existing under agent\bin folder. The details of how you can add this exclusion is described in KB0867749 When you define this exclusion make sure You use the complete path for wrapper-windows-x86-64.exe under agent folder. For example for a MID Server with the agent folder path C:\Midservers\mid1\agent you need to exclude "C:\Midservers\mid1\agent\bin\wrapper-windows-x86-64.exe" You mark “Apply to child processes” when you define the exclusion Add a new "File Scan" exclusion to the exclusion set for java.exe. This File Scan exclusion is defined to exclude the java.exe existing under agent\jre\bin folder. The details of how you can add this exclusion is described in KB0867749 When you define this exclusion make sure You use the complete path for java.exe under agent folder. For example for a MID Server with the agent folder path C:\Midservers\mid1\agent you need to exclude "C:\Midservers\mid1\agent\jre\bin\java.exe" You mark “Apply to child processes” when you define the exclusion Add a wildcard exclusion for the MID Server folder. For example for the agent folder "C:\Midservers\mid1\agent" you can add "C\Midservers\*". The details are available in KB0867749 . Add the new exclusion set to the policy applied for the MID Server host machine. Make sure the policy is synchronized on the MID Server host machine before starting MID upgrade. If you are not familiar with this process refer to KB0867749 . Cause the MID Server to upgrade.