MID Pre-Upgrade checks don't confirm there is sufficient Disk space for Download/Extraction/Rollback files before committing to upgrading - 'No space left on device' - Known Error

MID Pre-Upgrade checks don't confirm there is sufficient Disk space for Download/Extraction/Rollback files before committing to upgrading - 'No space left on device' Description If pre-checks are disabled, or if the 2GB free we check for isn't enough, MID Servers can run out of Disk Space while upgrading, which is something that should always be checked for before committing to an upgrade.

This probably won't leave the MID Severs Down or corrupted: - If this happens while downloading or extracting the ZIP files, the upgrade will abort before launching the upgrade process and shutting down the mid server - If this happens when running the upgrade process, during the backing up of files for Rollback, or copying new files over old, then Rollback should preserve the MID Server files as they were, and restart the MID Server service successfully on the previous version.

Steps to Reproduce

There are several requirements for a sizable amount of disk space during a MID Server upgrade: ~330MB - The ZIP files in agent/package/incoming/ ~390MB - Those files extracted into agent/work/upgrade_temp/ ~290MB - The rollback backups (the original folders are renamed as *_old, then new files copied back to the original folders). = over 1GB in total.

Actual Behaviour:

At any point the drive may run out of space, breaking the upgrade.

e.g. Running out of space while extracting the ZIP files looks like this in the Agent log, and despite this it keeps trying to extract more:

... INFO (AutoUpgrade.3600.now) [Packages:124] Packages refreshed. ... INFO (AutoUpgrade.3600.now) [MIDUpgrader:233] Upgrading MID server. ... INFO (AutoUpgrade.3600.now) [MIDUpgrader:440] Extracting package /xxx/midserver/agent/package/incoming/mid-upgrade.xanadu-07-02-2024__patch10-hotfix1-09-19-2025_09-21-2025_2155.universal.universal.zip to /xxx/midserver/agent/work/upgrade_temp/11160665709654973924 ... INFO (AutoUpgrade.3600.now) [MIDUpgrader:440] Extracting package /xxx/midserver/agent/package/incoming/mid-core.xanadu-07-02-2024__patch10-hotfix1-09-19-2025_09-21-2025_2155.universal.universal.zip to /xxx/midserver/agent/work/upgrade_temp/11160665709654973924 ... INFO (AutoUpgrade.3600.now) [ZipUtil:147] Extracting: /xxx/midserver/agent/work/upgrade_temp/11160665709654973924/agent ... ... INFO (AutoUpgrade.3600.now) [ZipUtil:147] Extracting: /xxx/agent/work/upgrade_temp/11160665709654973924/agent/lib/zstd-jni.jar ... ERROR (AutoUpgrade.3600.now) [ZipUtil:171] No space left on device java.io.IOException: No space left on device at java.base/java.io.FileOutputStream.writeBytes(Native Method) at java.base/java.io.FileOutputStream.write(FileOutputStream.java:349) at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:127) at com.glide.util.ZipUtil.unzip(ZipUtil.java:160) at com.glide.util.ZipUtil.unzip(ZipUtil.java:87) at com.service_now.mid.dist.upgrade.MIDUpgrader.extract(MIDUpgrader.java:447) at com.service_now.mid.dist.upgrade.MIDUpgrader.upgrade(MIDUpgrader.java:244) at com.service_now.monitor.AutoUpgrade.attemptUpgrade(AutoUpgrade.java:357) at com.service_now.monitor.AutoUpgrade.run(AutoUpgrade.java:177) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:275) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516) ... INFO (AutoUpgrade.3600.now) [ZipUtil:147] Extracting: /xxx/midserver/agent/work/upgrade_temp/11160665709654973924/agent/midinstaller ... INFO (AutoUpgrade.3600.now) [ZipUtil:147] Extracting: /xxx/midserver/agent/work/upgrade_temp/11160665709654973924/agent/midinstaller/checkedbox.png ... ERROR (AutoUpgrade.3600.now) [ZipUtil:169] /xxx/midserver/agent/work/upgrade_temp/11160665709654973924/agent/midinstaller/checkedbox.png (No such file or directory) java.io.FileNotFoundException: /xxx/midserver/agent/work/upgrade_temp/11160665709654973924/agent/midinstaller/checkedbox.png (No such file or directory) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293) at java.base/java.io.FileOutputStream. (FileOutputStream.java:235) at java.base/java.io.FileOutputStream. (FileOutputStream.java:184) at com.glide.util.ZipUtil.unzip(ZipUtil.java:157) at com.glide.util.ZipUtil.unzip(ZipUtil.java:87) at com.service_now.mid.dist.upgrade.MIDUpgrader.extract(MIDUpgrader.java:447) at com.service_now.mid.dist.upgrade.MIDUpgrader.upgrade(MIDUpgrader.java:244) at com.service_now.monitor.AutoUpgrade.attemptUpgrade(AutoUpgrade.java:357) at com.service_now.monitor.AutoUpgrade.run(AutoUpgrade.java:177) at com.snc.midserver.monitor.internal.MonitorRunner$MonitorTask.execute(MonitorRunner.java:275) at com.snc.midserver.monitor.internal.AMonitorTask.run(AMonitorTask.java:29) at java.base/java.util.TimerThread.mainLoop(Timer.java:566) at java.base/java.util.TimerThread.run(Timer.java:516)

e.g. Being unable to write the file for tracking the PIDs of the current java and wrapper processes looks like this in wrapper.log

... | ERROR: Could not write pid file /xxx/midserver/agent/work/mid.pid: No space left on device

e.g. While copying files back to original locations after renameing the original folders as the backup folders for use in Rollback. the warnings look like this in wrapper.log/dist_upgrade.log.0

INFO | jvm 1 | ... | Sep 17, 2025 1:03:52 PM com.snc.dist.mid_upgrade.UpgradeMain run INFO | jvm 1 | ... | INFO: The Upgrade rollback property has been set to true. Hence executing the new upgrade process ... INFO | jvm 1 | ... | Sep 17, 2025 1:03:52 PM com.snc.dist.mid_upgrade.UpgradeMain renameDirectoryWithRetry INFO | jvm 1 | ... | INFO: Successfully renamed the folder `/xxx/midserver/agent/lib` to `/xxx/midserver/agent/lib_old` INFO | jvm 1 | ... | Sep 17, 2025 1:03:52 PM com.snc.dist.mid_upgrade.UpgradeMain withRetry INFO | jvm 1 | ... | INFO: [Copying files from /xxx/midserver/agent/lib_old to /xxx/midserver/agent/lib] Attempt (1/5) INFO | jvm 1 | ... | Sep 17, 2025 1:03:55 PM com.snc.dist.mid_upgrade.UpgradeMain withRetry INFO | jvm 1 | ... | WARNING: [Copying files from /xxx/midserver/agent/lib_old to /xxx/midserver/agent/lib] failed with error: /xxx/midserver/agent/lib_old/closure-compiler.jar -> /xxx/midserver/agent/lib/closure-compiler.jar: No space left on device. Retrying... ... INFO | jvm 1 | ... | INFO: Failed to take the backup of /xxx/midserver/agent/lib INFO | jvm 1 | ... | Sep 17, 2025 1:11:55 PM com.snc.dist.mid_upgrade.UpgradeMain backupTheUpgradeFolders INFO | jvm 1 | ... | SEVERE: [Copying files from /xxx/midserver/agent/lib_old to /xxx/midserver/agent/lib] failed after 5 attempts. Error: /xxx/midserver/agent/lib_old/closure-compiler.jar -> /xxx/midserver/agent/lib/closure-compiler.jar: No space left on device INFO | jvm 1 | ... | com.snc.dist.mid_upgrade.exceptions.RetryOperationException: [Copying files from /xxx/midserver/agent/lib_old to /xxx/midserver/agent/lib] failed after 5 attempts. Error: /xxx/midserver/agent/lib_old/closure-compiler.jar -> /xxx/midserver/agent/lib/closure-compiler.jar: No space left on device INFO | jvm 1 | ... | at com.snc.dist.mid_upgrade.UpgradeMain.withRetry(UpgradeMain.java:1752) INFO | jvm 1 | ... | at com.snc.dist.mid_upgrade.UpgradeMain.backupTheUpgradeFolders(UpgradeMain.java:648) INFO | jvm 1 | ... | at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:382) INFO | jvm 1 | ... | at java.base/java.lang.Thread.run(Thread.java:840) INFO | jvm 1 | ... | Caused by: java.nio.file.FileSystemException: /xxx/midserver/agent/lib_old/closure-compiler.jar -> /xxx/midserver/agent/lib/closure-compiler.jar: No space left on device INFO | jvm 1 | ... | at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) INFO | jvm 1 | ... | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) INFO | jvm 1 | ... | at java.base/sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:256) INFO | jvm 1 | ... | at java.base/sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:603) INFO | jvm 1 | ... | at java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:257) INFO | jvm 1 | ... | at java.base/java.nio.file.Files.copy(Files.java:1305) INFO | jvm 1 | ... | at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:850) INFO | jvm 1 | ... | at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1312) INFO | jvm 1 | ... | at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:699) INFO | jvm 1 | ... | at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:630) INFO | jvm 1 | ... | at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:531) INFO | jvm 1 | ... | at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:502) INFO | jvm 1 | ... | at com.snc.dist.mid_upgrade.UpgradeMain.lambda$backupTheUpgradeFolders$0(UpgradeMain.java:649) INFO | jvm 1 | ... | at com.snc.dist.mid_upgrade.UpgradeMain.withRetry(UpgradeMain.java:1744) INFO | jvm 1 | ... | ... 3 more INFO | jvm 1 | ... | INFO | jvm 1 | ... | Sep 17, 2025 1:11:55 PM com.snc.dist.mid_upgrade.UpgradeMain backupTheUpgradeFolders INFO | jvm 1 | ... | INFO: Restoring the successfully backed up folders

Expected behaviour:

We know roughly how much disk space will be required, and so should be checking for at least that plus a safety margin as part of the pre-upgrade checks, _before_ committing to downloading/extracting the big ZIP files, and shutting down the MID Server and changing files. A clear Issues record reporting the lack of required disk space to the customer is required. Leaving the MID Server up in Upgrade Failed state is preferable to risking leaving the MID Server Down or in a semi-corrupted state.

Workaround This problem is currently under review and targeted to be fixed in a future release. Subscribe to this Known Error article to receive notifications when more information will be available.

This is not the root cause of a failed upgrade, and is more of a supportability enhancement to improve errors and reduce the risks of shutting down the mid server and copying files in the upgrade process before realising we are out of disk space.

The correct solution is to provision MID Server hosts with at least 50 GB of spare disk space. ECC Sender folders, Web server static files, synched files etc. do take up a considerable amount of disk space even when an Upgrade is not happening.