log4j found on a MID Server host, even after upgrading the Instance - Support and Troubleshooting

log4j found on a MID Server host, even after upgrading the Instance Issue This article is for confirming your active MID Servers are running the correct patched versions of Log4j and OpenJDK for the instance version, and finding old installs or backup folders that may still have old versions of files in, so that you can then uninstall/delete them.

Table of Contents Detailed information on MID Servers and Log4j Vulnerabilities Expected log4j, SLF4J, Reload4j, and OpenJDK versions for recent releases, and old left-over files Confirming JAR file versions Confirming OpenJDK version Finding all active MID Servers Finding Services and Files Deleting/Uninstalling MID Server files Detailed information on MID Servers and Log4j Vulnerabilities This article is deliberately not going into the potential vulnerabilities. See KB1000959 for details.

Expected log4j, SLF4J, Reload4j, and OpenJDK versions for recent releases, and old left-over files If you have upgraded the instance, and the MID Servers have been able to upgrade themselves to match, then these are the file versions you should have now.

At the time of writing (autumn 2022), all instances should have been upgraded to the 2.17.1 version of log4j. This list is mainly so you can confirm which ServiceNow versions old version files might have come from, as part of confirming that they are from inactive installs.

Important Note: log4j-to-slf4j.jar and log4j-over-slf4j.jar are translation wrappers , to allow legacy code calls to log4j APIs to be redirected to SLF4J instead. These were used to allow the MID Server to no longer require Log4j. From the list below, you can see that jarlog4j-to-slf4j.jar replaced the problematic log4j-core.jar, which contains the same functions, so dependencies are not broken, but they all redirect to SLF4J instead.

Note on Reload4j: The reload4j project is a fork of Apache log4j.jar version 1.2.17, and a drop-in replacement of log4j.jar, in order to fix most pressing security issues.

ServiceNow Version Log4j files SLF4J files Reload4j files Java files log4j.jar log4j-core.jar log4j-to-slf4j.jar log4j-slf4j-impl.jar log4j-api.jar log4j-over-slf4j.jar reload4j.jar OpenJDK Calgary, Dublin, Eureka, Fuji, Geneva, Helsinki, Istanbul, Jakarta 1.2.8, 1.2.15, 1.2.17 Not included Not included Not included Not included Not included Not included 6u14, 6u45, 8u40, 8u60 Kingston, London 1.2.17 Not included Not included Not included Not included Not included Not included 1.8.0_152 Madrid, New York, Orlando 1.2.17 Not included Not included Not included Not included Not included Not included 1.8.0_181 Paris, Orlando Patch 3, New York Patch 8 1.2.17 Not included Not included Not included Not included Not included Not included 1.8.0_232 Quebec 1.2.17 Not included Not included Not included 2.11.1 Not included Not included 11.0.8 Quebec Patch 10, Quebec Patch 9 Hot Fix 3, Not included 2.16.0 Not included 2.16.0 2.16.0 1.7.30 Not included 11.0.8 Quebec Patch 10 HotFix 1, Quebec Patch 9 Hot Fix 4a, Quebec Patch 9 Hot Fix 3b Not included 2.17.1 Not included 2.17.1 2.17.1 1.7.30 1.2.18.2 11.0.8 Rome Not included 2.14.0 Not included 2.14.0 2.14.0 1.7.30 Not included 11.0.9.1 Rome Patch 6, Rome Patch 5, Rome Patch 4 Hot Fix 1 Not included 2.16.0 Not included 2.16.0 2.16.0 1.7.30 Not included 11.0.9.1 Rome Patch 7, Rome Patch 6 Hot Fix 1, Rome Patch 4 Hot Fix 1b Not included 2.17.1 Not included 2.17.1 2.17.1 1.7.30 1.2.18.2 11.0.9.1 San Diego Not included 2.16.0 Not included 2.16.0 2.16.0 1.7.30 Not included 11.0.12 San Diego Patch 1 Not included 2.17.1 Not included 2.17.1 2.17.1 1.7.30 1.2.18.2 11.0.12 Tokyo Not included Not included 2.17.1 Not included 2.17.1 1.7.36 Not included 11.0.15 Utah Not included Not included 2.17.1 Not included 2.17.2 1.7.36 Not included 11.0.16.1 Vancouver Not included Not included 2.17.1 Not included 2.17.2 1.7.36 Not included 17.0.5 Only the first major versions release, and patch versions where things changed are shown.

Confirming JAR file versions You can manually confirm the JAR file versions by unzipping the jar file, e.g. agent]\lib\log4j-core.jar , and opening the META-INF\MANIFEST>MF file within. Note: JAR files are like ZIP files, and can be opened by e.g. 7-Zip.

Confirming OpenJDK version You can confirm the OpenJDK Java Runtime Environment version, by opening the agent\jre\release file .

Finding all active MID Servers In the instance, open a list of MID Servers. Add the " Host name " and " Home folder " column to the list. These columns will tell you where the install folder is.

If the MID Server is Up then it is active, and should be on the same version as the instance. If it is Up and on an older version, ithe reason for the failed upgrade should be investigated and solved, because it needs to be upgraded to be compatible with the instance. If it is in Down status , but the Last Refreshed date is very recent, then the MID Server is probably still active, but has recently had a communication problem that needs investigating and solving. If a MID Server is Down , and has been down for a long while, where the Last Refreshed date may be months or years ago, then you can assume this MID Server is no longer in use. Only a MID Server that is Up will know that it has to upgrade, so often Down MID Servers will still be on the old version they were running when they were last used. If MID Server records in the instance are deleted , this does not actually uninstall the MID Server from the host server it was originally installed on. You may have lost track of some old installs due to having done this. The home folder is the 'agent' folder . Within that are the agent\lib and agent\jre folders, that any scan results' files are likely to be in. If your scanner reports files in folders that you cannot account for from the lists of mid server sin your instances, then they are probably inactive installs, or perhaps backup folders of installs, or temporary folders created while fixing something in the past. Those can all be deleted. Finding Services and Files If you know the host, but not much else, you can find Services with the command line: wmic service get | findstr /c:"\conf\wrapper.conf" /c:"DisplayName"

To find the files that may be an old backup folder, or temp folders, and not actually installed as services, you can use dir: dir /s /b mid.jar

Comparing the UP mid server record, with those active services, and those folders, will tell you what can be deleted. You may choose to use a spreadsheet to keep track of these and match them up.

Deleting/Uninstalling MID Server files Once you have confirmed an install folder ('agent' folder) is not for a currently active MID Server, you can remove the windows service and delete the files.

Don't forget that you have at least 2 instances, and so you should compile a list of all active MID Server host name/home folder, from all instances, before deciding to start deleting anything inactive.

Stop the windows service. The command line "wmic service get | findstr /c:"\conf\wrapper.conf" /c:"DisplayName" will output all MID Server services. The "Name" field is the service name, and "PathName" will allow you to confirm which install folder this service is for, to match to your scan results. Delete the windows service. Delete the service with the command line "sc delete " Delete the files. The home folder/"agent" folder and everything within can be deleted. Delete the MID Server record in the instance Related Links KB1536827 Listing of MID Server lib folder JAR File manifests including Version information (Vancouver EA)

KB1184230 Microsoft Defender gives false alert for Log4j exploitation detected/Apache exploitation detected and Multi-stage incident on MID Server host