How to minimise the impact of the MID Servers momentarily going down while upgradingIssue Immediately after an Instance upgrade (including patches, hotfixes, and weekly upgrades), all MID Servers will also Auto-Upgrade themselves to match the new version, otherwise there may be code and data mismatches which can break things. During that auto-upgrade, the MID Servers will momentarily go DOWN in order to replace application files before restarting. That can take approx 10 seconds to 2 minutes depending on the performance of the host server, and the time spend waiting for file locks to clear. This is almost always not noticed and shouldn't cause any issues, because: by its nature MID Server is a fundamentally Async feature that works via an ECC Queue, with Sensor businss rules to respond and continue processing when the result is returned.Jobs waiting in the ECC Queued, will remain queued, and run once the MID Server is back up, as per the normal queue rules.Already running jobs will have enough time to finish before the MID Server is restarted, and if they don't then they will be re-run on startup automatically This could in theory only cause jobs to fail, if: A job is sent to a specific MID Server, isand has a timeout set in the instance. e.g. Synchronous RESTMessageV2 is set to 15s timeout, and the MID Server hasn't a chance to run it in that time.A job uses MID Selection algorithms, and all the MID Servers that match are not back up after the upgrade yet. MID Server Down and Up events will be caused, which may lead to Notifications being sent if those notifications have been configured with any users. Long-running jobs may get interrupted, and so will run again from the beginning after the upgrade. For a data push, that may theoretically cause duplicate inserts/updates at the endpoint. A job that has already finished may run again if the MID Server is restarted after finishing a job, but the input has not yet been passed back to the instance. See KB0743121/PRB1330405 for more details. Features always affected by a restart: These features need a single dedicated MID Server, so will be affected when that MID Server restarts: LDAPExport setsJDBC data sourcesSOAPMessageV2/RESTMessageV2 API called synchronously from scripts, without a sensor business rule.and probably others. Features not affected by the restart: Discovery / Service Mapping - Probes/Sensors will continue to run after the upgrade. Any interrupted will re-run.Orchestration - Assuming more than 1 MID Server is set up for each combination of Capability/Application/IP Range, and at least one remains UP. If no MID Server is available then the workflow Activity will error, and transition to the error, and assuming the workflow is designed to loop back and retry after a short timer, then this is not an issue. and probably others. Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Anywhere MID Servers are in use. However this has become more of an issue since 2026, when weekly instance upgrades are now happening, meaning MID Server have to upgrade a lot more to match. Cause This is the expected behavior and unavoidable due to the current design of the MID Server and its upgrade code: Files cannot be replaced with new versions while they are in use. It has become more of an issue since mid-2026 when weekly instance patching started. Resolution There are several things that can be done to minimize the risk, depending on how the MID Server is used. A lot of these would also be useful when a specified MID Server is down, perhaps due to an issue unrelated to an upgrade. Table of Contents Avoid running more than you have to around the time of an Instance UpgradeConfigure Fail-over Clusters or MID SelectionRetrofit an Integration to use MID SelectionMake use of the "Fail over MID server" script action, for jobs already queued in Ready stateControl when the Upgrade runs, by temporarily Pinning the MID Server versionPin a single member of each MID Server cluster until the other members are upgradedMinimize the number of MID Servers that can Upgrade at the same time to OneSkip some upgradesConfigure Integration Hub Connections to use MID Server auto-select, with a custom MID Server application, to get Load Balancing and Failover for free Avoid running more than you have to around the time of an Instance Upgrade During the instance upgrade, the system scheduler is paused, so that most scheduled jobs will not run. At the end of the instance upgrade, this is enabled again, and that is before the MID Servers start to upgrade, so it is possible a lot of queued jobs are likely to be trying to run at exactly the same time as the MID Servers are starting their auto-upgrades. Avoid that my manually disabling, or deactivating any MID Server-related jobs before the upgrade, and only turn them back on again after the MID Servers have also upgraded. Long-running jobs may get interrupted, and so will run again from the beginning after the upgrade. For a data push, that may theoretically cause duplicate inserts/updates at the endpoint. A backlog may build up in the ECC queue or System scheduler tables during the instance upgrade, that then runs all at once after the upgrade. Those kind of spikes can be problematic for MID Server performance. Configure Fail-over Clusters or MID Selection Depending on a single named MID Server is a risk. Where possible, select MID Server automatically, via MID Selection, Discovery Behaviours or Clusters. If one MID Server is DOWN then another can be used. Not all features are yet updated to leverage these MID Server features. Add your MID Server and additional MID Servers to a Cluster configured for Fail-over protection, and configure the fail-over MID Servers with at least the same capabilities as the MID Server it is intended to relieve. Note: More than one MID Server can be installed on the same Host server. A typical fail-over cluster might be 2 MID Servers each on 2 Host servers, totaling 4 MID Servers. Retrofit an Integration to use MID Selection When using RESTMessageV2 or SOAPMessageV2, which don't natively support MID Selection or Clusters, the MID Selector APIs can be used to select the MID Server, which is then passed to the RESTMessageV2/SOAPMessageV2 APIs. Two examples showing ways to do this: IP + Capability var mss = new MIDServerSelector();var capab = ["ssh","wmi","snmp"];mss.setCapabilities(capab);mss.findAgent('100.101.10.10'); IP + Capability + Application, which could be a custom Application value specific to the integration var midSelector = new SNC.MidSelector();gs.info ( midSelector.selectMid("Orchestration", ['10.100.99.100'], [{"capability": "test2"}, {"capability": "test1"}])) ; Make use of the "Fail over MID server" script action, for jobs already queued in Ready state When a mid_server.down event is fired, which is a record in the sysevent table create when any MID Server status is set as Down, a Script Action called 'Fail over MID server' runs. When a MID Server momentarily goes down during a restart, that script action will run. This will re-assign any ECC Queue output records still in Ready or Processing state to another MID Server, using either of these methods: If the job was assigned to a MID Server using MID Selector in the first place: Re-run the MID Selection algorithms using the same criteria as the original MID Selection was done.This depends on another MID Server still being up, with the same Capabilities/IP Ranges/Applications defined as the down MID Server If the job was assigned to a MID Server directly, or via a Cluster: Assign the job to another member of the same Failover Cluster that the MID Server is a member of.This depends on the MID Server being a member of a Failover Cluster, and there being other members that are up. This will also re-assign outputs that are already in processing state, which guarantees all jobs get run, but can mean the job ends up getting run twice: 'Processing' means the mid server already took the jobs from the queue. It may be at any of these stages of the process: The MID Server has the job in its internal queue on RAM. (MID Servers deliberately fetch more jobs at a time than they have available threads to run them in)The job may be executing in a worker threadThe job may have completed, and the result saved in a temporary XML file in the ECC Sender Folder, but not yet returned to the instance as an ECC Input record.The ECC input record was inserted, but the Business Rules that set the Output record to Processed in response to that have not done that yet. If the MID Server is only temporarily down, due to a network/communication issue between the MID Server and Instance, but is actually still running, this can lead to jobs that were reassigned in Processing state to be run by the original and failover MID Server. It is rare but does happen. Some integrations can have a problem with this, such as records being pushed to another system twice causing duplicates. There is a downside to using a legacy Load Balancing MID Server cluster, as it is implemented by a before insert businss rule, that does not take into account which Cluster was used initially. It only takes into account the MID Server initially set in the record. This means it could be reassigned to a MID Server not in that original cluster, but that was in a different cluster the original MID Server was also in (even though that other cluster wasn't the one configured in the applciation). Control when the Upgrade runs, by temporarily Pinning the MID Server version If a MID Server is Pinned, then it will stay on that version. This can be used to delay the upgrade until you are sure no jobs will be affected while it happens. Warning: The auto-upgrade should not be delayed more than a few minutes due to the risk of code and data mismatches breaking things while the MID Server version is wrong for the instance. For each MID Server:- add the Parameter mid.pinned.version and set the value as the MID buildstamp: value shown in your Stats page (/stats.do) before the upgrade.e.g. for an upgrade from London Patch 4 to Madrid: Name: mid.pinned.versionValue: london-06-27-2018__patch4-11-21-2018_12-04-2018_1527Allow the instance upgrade to finish.Ensure no jobs will need to use the MID Server in the next few minutes.Delete the mid.pinned.version parameter. (Delete the parameter record. Don't just clear it, or it will not be un-pinned)Click Upgrade on the MID Server form to cause the Upgrade to happen now. Pin a single member of each MID Server cluster until the other members are upgraded You can temporarily pin just one of the MID Servers in each fail-over cluster, to ensure one will remain up for whatever jobs are en-queued during the upgrades of the other cluster members. Minimize the number of MID Servers that can Upgrade at the same time to One Warning: This feature is considered deprecated, since it caused many performance issues when API Semaphores of the instance were being blocked by many concurrent large downloads from MID Servers. This in turn blocked other MID Servers from connecting at all, leaving them down, but also blocked inbound integrations for features such as Event Managmeet. Since the London release, it is possible to restrict the number of concurrent MID Server upgrades. This is designed to avoid SOAP semaphore exhaustion but has the side-effect of allowing you to control the maximum number of MID Servers that can be downloading the upgrade files at the same time. This should allow the upgrades of MID Servers to be spread out more, however doesn't guarante it for mid servers with a fast iternet connection tot he instance, as the download phase is only the start of the upgtrade. All MID Servers could still end up in the hfile copy phase at the same time. Add these two properties to the instance System Property [sys_properties] table: mid.download.through.instance=trueconcurrent.dist.download=1 Skip some upgrades At the time of writing, all instance version changes will cause MID Server upgrades. e.g. Patch 5 week 39 to Patch 5 week 40 will still trigger an upgrade, even though no MID Server-related code has changed. All MID Servers could be pinned, and kept pinned until the next instance upgrade, after reviewing whether the next upgrade is needed. However this is dangerous as some changes to dependencies do happen in patches and hotfixes, that do affect MID Server, and do break things when the MID Server remains on the previous version. e.g. It is well known that Integtration Hub probes check the mid server version, and error if the major version is mismatched, but there are also dependency errors being seen in Integration when the instance is Australia Patch 5, but the MID Server remains patch 4. So when the banner along the top of the MID Server form states the version is mismatched, but 'compatible', that isn;t necessarily true, depending on which feature's code it is. There is lots of Java in the mid server application that is from individual higher-level apps. For example Event Management's Web server. Discovery/Service Mapping's Pattern Engine. ACC's Endpoint extension. Our MID Server platform developers don't own that code, and don't do release testing with it, so when they say Compatible then meant he MID Server platform in general, not that higher level code. Configure Integration Hub Connections to use MID Server auto-select, with a custom MID Server application, to get Load Balancing and Failover for free Connections for Integration Hub Flow steps can be set to use a specific MID Server, MID Server load balancing cluster, or use Auto-select. Setting a specific MID Server has all your eggs in one basket. Using a Load Balancing Cluster is not ideal either, as it only gives a MID Server you expect if that MID Server is only in a single Load Balancing Cluster. The best solution is to create a new ecc_agent_application record for your Step, Flow or set of similar flows. That Application would then be set in your Connection record. The Application would then be assigned to the MID Servers that can run those steps, making a set of MID Server that will work in a way similar to Load Balancing plus Fail-over cluster, but without any of the downsides and legacy code of MID Server Clusters. Related Links<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Main Docs page for: MID Server UpgradeMID Server Cluster ConfigurationMID Server selection