File permissions enforcement removes permissions of the windows service's log on as user, when computer or domain name has lowercase letters, leaving MID Server down


Description

Windows MID Server file permissions enforcement removes folder permissions for the Windows service's "log on as" user when the computer or domain name has lowercase letters. Only the System user and Administrators group will still have access.
After upgrade to Orlando or later, the ACL for the non-admin log on as user gets removed from the folder permissions. This prevents the MID Server from operating correctly after the permissions have been changed (e.g. probes can't write results to ECCSender folders for ECCSender to send back to the instance), and prevents the service from starting at all when the MID Server is restarted ("Windows could not start the service Error 5: Access is denied").

In an upgrade situation, this will leave the MID Server Down, and cause an outage for any services and integrations using the MID Server.

A new install using the Paris MSI installer can have the same issue, as a non-admin user will always be used. In this case, the MID Server record [ecc_agent] in the instance never gets created.

Steps to Reproduce

  1. Have a windows host with a hostname that is not completely capitalized.  Ex: Rather than "WIN-HOST-NAME" it would be something like "Win-Host-Name", or any non capitalized letters.
  2. Install the mid service, and set the log on user as a non-admin service account. (along with ACLs allowing the non-admin account full access to the agent folder and contained files, can use the the msi installer for this)
  3. Attempt to run the mid server.

Expected:  MID Server works normally, can start up and does not run into ACL issues.
Actual: During MID startup, the ACL for non-admin user gets removed, and they no longer have access to MID Folder.  This prevents the MID from starting.

The agent log for a new install, or after the upgrade, will appear something like (this is taken from a fresh MSI install, on a windows host with name including lower case letters):

...
09/18/20 03:07:07 (158) MIDServer MID Server started
09/18/20 03:07:07 (164) StartupSequencer PowerShell path not configured, resolving...
09/18/20 03:07:07 (164) StartupSequencer     searching for PowerShell executable "powershell.exe"...
09/18/20 03:07:07 (633) StartupSequencer     found PowerShell executable at "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
09/18/20 03:07:07 (648) StartupSequencer Running command to determine Powershell version: "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -noninteractive -nologo -noprofile -command "$ver = if (Test-Path Variable:\PSVersionTable) { $PSVersionTable.PSVersion } else { (get-host).Version }; 'full_version:' + $ver.ToString() + ', major_version:' + $ver.Major"
09/18/20 03:07:15 (492) StartupSequencer     PowerShell version result: full_version:5.1.17763.316, major_version:5
09/18/20 03:07:15 (492) StartupSequencer     verify Powershell major version 5 against compatible version requirement (v3 - v5)
09/18/20 03:07:15 (492) StartupSequencer PowerShell path is set to "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe", version 5.1.17763.316
09/18/20 03:07:15 (492) StartupSequencer DEBUG: HTTPClient.registerOtherProtocols() starting on Thread Thread[StartupSequencer,5,main].
09/18/20 03:07:31 (384) StartupSequencer The service name is detected as snc_mid_VB MSI PreGA
09/18/20 03:07:35 (602) StartupSequencer WARNING *** WARNING *** Continuing with start up, but Windows file system permissions enforcer encountered an issue: icacls : Permission denied
At line:1 char:1
+ icacls "C:\ServiceNow MID Server VB MSI PreGA\agent" /save "C:\Servic ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Permission denied:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

09/18/20 03:07:35 (617) StartupSequencer WARNING *** WARNING *** Unable to log the following MID Issue due to unknown MID sys_id: An unexpected error occurred: icacls : Permission denied
At line:1 char:1
+ icacls "C:\ServiceNow MID Server VB MSI PreGA\agent" /save "C:\Servic ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Permission denied:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 

09/18/20 03:07:35 (617) StartupSequencer DEBUG: HTTPClient.registerOtherProtocols() starting on Thread Thread[StartupSequencer,5,main].
09/18/20 03:07:36 (061) StartupSequencer Successfully connected to instance:
09/18/20 03:07:36 (061) StartupSequencer     Install name: Demo Server
09/18/20 03:07:36 (061) StartupSequencer    Instance name: empdpiper
09/18/20 03:07:36 (061) StartupSequencer             Node: e986ad080b6baf010dd60f6c919d719f
09/18/20 03:07:36 (061) StartupSequencer       Build date: 09-17-2020_2030
09/18/20 03:07:36 (061) StartupSequencer        Build tag: glide-paris-06-24-2020
09/18/20 03:07:36 (061) StartupSequencer      Instance ID: 5ebbe8999cb80dc09bd847c7b6052a29
09/18/20 03:07:36 (071) StartupSequencer        System ID: app135023.ytz3.service-now.com:empdpiper013
09/18/20 03:07:36 (071) StartupSequencer      Instance IP: 10.87.135.23
09/18/20 03:07:36 (071) StartupSequencer      MID buildstamp: paris-06-24-2020_09-17-2020_2030
09/18/20 03:07:36 (071) StartupSequencer DEBUG: HTTPClient.registerOtherProtocols() starting on Thread Thread[StartupSequencer,5,main].
09/18/20 03:07:36 (414) StartupSequencer SOAP basic authentication is enabled
09/18/20 03:07:36 (414) StartupSequencer SOAP strict security is enabled
09/18/20 03:07:36 (414) StartupSequencer User mid_user has all necessary roles
09/18/20 03:07:36 (414) StartupSequencer Verifying read access to every directory and file in tree C:\ServiceNow MID Server VB MSI PreGA\agent
09/18/20 03:07:36 (414) StartupSequencer SEVERE *** ERROR *** Cannot read directory: C:\ServiceNow MID Server VB MSI PreGA\agent, reading the directory failed with the exception java.nio.file.AccessDeniedException: C:\ServiceNow MID Server VB MSI PreGA\agent
09/18/20 03:07:36 (430) StartupSequencer SEVERE *** ERROR *** test failure
java.lang.IllegalStateException: Filesystem permissions are incorrect
 at com.service_now.mid.services.StartupSequencer.runTests(StartupSequencer.java:572)
 at com.service_now.mid.services.StartupSequencer.startupSequencerRunnable(StartupSequencer.java:611)
 at java.lang.Thread.run(Thread.java:748)

09/18/20 03:08:36 (438) StartupSequencer DEBUG: HTTPClient.registerOtherProtocols() starting on Thread Thread[StartupSequencer,5,main].
09/18/20 03:08:37 (140) StartupSequencer Successfully connected to instance:
09/18/20 03:08:37 (140) StartupSequencer     Install name: access restricted
09/18/20 03:08:37 (140) StartupSequencer    Instance name: access restricted
09/18/20 03:08:37 (140) StartupSequencer             Node: access restricted
09/18/20 03:08:37 (140) StartupSequencer       Build date: access restricted
09/18/20 03:08:37 (140) StartupSequencer        Build tag: access restricted
09/18/20 03:08:37 (140) StartupSequencer      Instance ID: access restricted
09/18/20 03:08:37 (140) StartupSequencer        System ID: access restricted
09/18/20 03:08:37 (140) StartupSequencer      Instance IP: access restricted
09/18/20 03:08:37 (140) StartupSequencer      MID buildstamp: paris-06-24-2020_09-17-2020_2030
09/18/20 03:08:37 (140) StartupSequencer DEBUG: HTTPClient.registerOtherProtocols() starting on Thread Thread[StartupSequencer,5,main].
...

Workaround

This problem has been fixed. If you are able to upgrade, review the Fixed In or Intended Fix Version fields to determine whether any versions have a planned or permanent fix.

As a workaround, the MID Server Parameter mid.windows_host.file_permissions.enforce=false will disable the script.


Related Problem: PRB1396279