Failure creating instance of ECCSenderCache, unable to continueIssue The MID Server may go Down, and keep restarting in a loop, with the following ECCSender related errors: 2023-01-31T16:22:12.463-0500 INFO (main) [ALoggerConfigProvider:56] Logger config: root=INFO2023-01-31T16:22:13.026-0500 INFO (WrapperStartStopAppMain) [Config:162] Using configuration: C:\ServiceNow\MidServer\agent\config.xml...2023-01-31T16:22:23.869-0500 INFO (MIDServer) [MIDServerState:94] MID Server starting...2023-01-31T16:23:01.510-0500 INFO (StartupSequencer) [Monitors:307] Enabling monitor: ECCQueueMonitor2023-01-31T16:23:01.510-0500 INFO (StartupSequencer) [SenderFilesInfoService:44] Service started.2023-01-31T16:23:01.525-0500 INFO (StartupSequencer) [ECCSenderCache:125] ECC Sender Cache initialization: step 1 of 4: setting up cache directories...2023-01-31T16:23:01.525-0500 INFO (StartupSequencer) [ECCSenderCache:133] ECC Sender Cache initialization: step 2 of 4: building list of sysids from each file in cache directories...2023-01-31T16:23:01.619-0500 INFO (StartupSequencer) [QueueProcessingTracker:168] Map of sys_ids initial size: 02023-01-31T16:23:01.822-0500 INFO (StartupSequencer) [MIDServerInfoRemote:250] Getting 3 ECC queue output record sys_ids for agent: mid.server.xxx2023-01-31T16:35:45.816-0500 INFO (WrapperListener_stop_runner) [Main:328] Stopping MID server It may get to step 3 before going wrong: 03/24/22 03:56:59 (994) StartupSequencer ECC Sender Cache initialization: step 3 of 4: for all ECC Queue output records assigned to this MID Server which are in Processing state: change state to Ready...03/24/22 03:57:00 (009) StartupSequencer Setting all ECC queue output records from processing state to ready for agent: mid.server.<mid server name> Meanwhile, the wrapper.log has: 2023/01/31 16:22:08 | --> Wrapper Started as Service ... 2023/01/31 16:23:02 | Exception in thread "StartupSequencer" com.google.inject.ProvisionException: Unable to provision, see the following errors:2023/01/31 16:23:02 | 2023/01/31 16:23:02 | 1) [Guice/ErrorInCustomProvider]: RuntimeException: Failure creating instance of ECCSenderCache, unable to continue2023/01/31 16:23:02 | at ECCSenderModule.provideEccSenderCache(ECCSenderModule.java:98)2023/01/31 16:23:02 | \_ installed by: MonitorsModule -> ECCSenderModule2023/01/31 16:23:02 | at ECCSenderModule.provideEccSender(ECCSenderModule.java:79)2023/01/31 16:23:02 | \_ for 2nd parameter2023/01/31 16:23:02 | at ECCSenderModule.provideEccSender(ECCSenderModule.java:79)2023/01/31 16:23:02 | \_ installed by: MonitorsModule -> ECCSenderModule2023/01/31 16:23:02 | while locating ECCSender2023/01/31 16:23:02 | at ECCSenderModule.configure(ECCSenderModule.java:66)2023/01/31 16:23:02 | \_ installed by: MonitorsModule -> ECCSenderModule2023/01/31 16:23:02 | while locating AMonitor annotated with @Element(setName=,uniqueId=10, type=MAPBINDER, keyType=String)2023/01/31 16:23:02 | 2023/01/31 16:23:02 | Learn more:2023/01/31 16:23:02 | https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER2023/01/31 16:23:02 | 2023/01/31 16:23:02 | 1 error2023/01/31 16:23:02 | 2023/01/31 16:23:02 | ======================2023/01/31 16:23:02 | Full classname legend:2023/01/31 16:23:02 | ======================2023/01/31 16:23:02 | AMonitor: "com.service_now.monitor.AMonitor"2023/01/31 16:23:02 | ECCSender: "com.service_now.monitor.ECCSender"2023/01/31 16:23:02 | ECCSenderModule: "com.snc.midserver.monitor.eccsender.internal.ECCSenderModule"2023/01/31 16:23:02 | Element: "com.google.inject.internal.Element"2023/01/31 16:23:02 | MonitorsModule: "com.snc.midserver.monitor.internal.MonitorsModule"2023/01/31 16:23:02 | ========================2023/01/31 16:23:02 | End of classname legend:2023/01/31 16:23:02 | ========================2023/01/31 16:23:02 | 2023/01/31 16:23:02 | at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)2023/01/31 16:23:02 | at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1103)2023/01/31 16:23:02 | at com.service_now.mid.services.Monitors.lambda$start$0(Monitors.java:74)2023/01/31 16:23:02 | at com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:185)2023/01/31 16:23:02 | at com.service_now.mid.services.Monitors.start(Monitors.java:73)2023/01/31 16:23:02 | at com.service_now.mid.services.Monitors.onMIDServerEvent(Monitors.java:429)2023/01/31 16:23:02 | at com.service_now.mid.services.Events.internalFire(Events.java:102)2023/01/31 16:23:02 | at com.service_now.mid.services.Events.fire(Events.java:34)2023/01/31 16:23:02 | at com.service_now.mid.services.StartupSequencer.startServices(StartupSequencer.java:402)2023/01/31 16:23:02 | at com.service_now.mid.services.StartupSequencer.testsSucceeded(StartupSequencer.java:179)2023/01/31 16:23:02 | at com.service_now.mid.services.StartupSequencer.startupSequencerRunnable(StartupSequencer.java:734)2023/01/31 16:23:02 | at java.base/java.lang.Thread.run(Thread.java:829)2023/01/31 16:23:02 | Caused by: java.lang.RuntimeException: Failure creating instance of ECCSenderCache, unable to continue2023/01/31 16:23:02 | at com.snc.midserver.monitor.eccsender.internal.ECCSenderModule.provideEccSenderCache(ECCSenderModule.java:111)2023/01/31 16:23:02 | at com.snc.midserver.monitor.eccsender.internal.ECCSenderModule$$FastClassByGuice$$1983005.GUICE$TRAMPOLINE(<generated>)2023/01/31 16:23:02 | at com.snc.midserver.monitor.eccsender.internal.ECCSenderModule$$FastClassByGuice$$1983005.apply(<generated>)2023/01/31 16:23:02 | at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:260)2023/01/31 16:23:02 | at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:171)2023/01/31 16:23:02 | at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)2023/01/31 16:23:02 | at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)2023/01/31 16:23:02 | at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)2023/01/31 16:23:02 | at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)2023/01/31 16:23:02 | at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)2023/01/31 16:23:02 | at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:40)2023/01/31 16:23:02 | at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:60)2023/01/31 16:23:02 | at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:171)2023/01/31 16:23:02 | at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)2023/01/31 16:23:02 | at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)2023/01/31 16:23:02 | at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)2023/01/31 16:23:02 | at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)2023/01/31 16:23:02 | at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)2023/01/31 16:23:02 | at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:60)2023/01/31 16:23:02 | at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)2023/01/31 16:23:02 | at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)2023/01/31 16:23:02 | at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)2023/01/31 16:23:02 | at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1100)2023/01/31 16:23:02 | ... 10 more2023/01/31 16:23:02 | Caused by: java.io.IOException: No 'sysIds' array returned in response.2023/01/31 16:23:02 | at com.service_now.mid.MIDServerInfoRemote.getEccQueueProcessingSysIds(MIDServerInfoRemote.java:284)2023/01/31 16:23:02 | at com.service_now.mid.Instance.getQueueProcessingSysIds(Instance.java:187)2023/01/31 16:23:02 | at com.service_now.monitor.ECCSenderCache.<init>(ECCSenderCache.java:170)2023/01/31 16:23:02 | at com.snc.midserver.monitor.eccsender.internal.ECCSenderModule.provideEccSenderCache(ECCSenderModule.java:108)2023/01/31 16:23:02 | ... 32 more2023/01/31 16:35:45 | Stopping the ServiceNow MID Server service...2023/01/31 16:35:50 | Waiting to stop...2023/01/31 16:35:50 | <-- Wrapper Stopped2023/01/31 16:35:52 | ServiceNow MID Server service stopped. CauseThere are several causes we know of: The file/folder permissions in the work\monitors\ECCSender folder are wrong, or the whole folder is missing.PRB1567004 MID server ECC sender cache initialization fails on startup when there are many processing ECC queue records to set back to 'Ready' Fixed since Tokyo.PRB1623655 Processing records count doesn't match sysIds count when invoked from ECCSenderCacheSeen only in San Diego.PRB1623622 MID Server MID server going down with the exception 'Failure retrieving sys IDs of ECC queue records in processing state' Being fixed as part of PRB1623655.PRB1636117 MID Server status is down during Cloud Discovery runs - Observed OOM Error (triggered by promotheus) & ECCSenderCache related exceptionUnconfirmed.PRB1579106 NPEs happening in rare case when MID has a lot files in ECCSender output folder Fixed in UtahResolutionConfirm the <install folder>\agent\work\monitors\ECCSender folder is present, and that file/folder permissions for the whole agent folder all subfolders and files are set correctly. Check what xml files exist in the output folder within the ECCSender folder. If there are thousands, that could be the cause, and could be moved out of there to allow the MID Server to start up.Check what ecc_queue output records exist for this MID Server in the instance. If there are thousands in ready or processing state, then they could be excluded from being picked up by the MID Server by setting the state to Error.Check that all the Scripted SOAP Services from the MID Server package are the correct out-of-box version, especially GetMIDInfo, and revert to out-of-box if they had been customised or skipped in an upgrade, or if an old version had been preserved in a clone from a newer source instance version.Check agent log for errors relating to GetMIDInfo or UpdateQueueFromProcessingToReady. There could be a timeout, or if there is an immediate error for that request and status 500, it could be due to special characters in the MID Server name, such as a unicode long hyphen. App node localhost logs will give more information.Remove mid.last.queried.date from (<MID agent directory>/work) folder. (if the file exists)Add the configuration parameter "mid.eccq.update.min.rank.query.win" and set as false in config.xml file. If all else fails, follow the procedure in: KB0713557 How to manually restore or upgrade a MID Server after a failed auto-upgrade