Semaphore Exhaustion and High Instance Response Time Due to Heavy Cache Flushing on the instance - Support and Troubleshooting

Semaphore Exhaustion and High Instance Response Time Due to Heavy Cache Flushing on the instance Issue ServiceNow internal monitoring system alerts when it detects semaphore exhaustion and elevated response times on your instance At this time you may also experience increased response times due to longer running transactions

You can find sys cache flush queries running on the Database at this time

Query 2611863 Writing to net abc appxxxx abc_1 14:28 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2611829 Writing to net abc appxxxx abc_1 14:05 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2608814 Writing to net abc appxxxx abc_1 13:19 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2611848 Writing to net abc appxxxx abc_1 12:06 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2611934 Writing to net abc appxxxx abc_1 12:06 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2610291 Writing to net abc appxxxx abc_1 11:04 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2610304 Writing to net abc appxxxx abc_1 09:50 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2610298 Writing to net abc appxxxx abc_1 09:43 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys Query 2610293 Writing to net abc appxxxx abc_1 09:29 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys

The logfiles will show these entries:

2019-07-15 09:02:23 (675) Default-thread-11 SYSTEM txid=18dfd39bdb22 Cache synchronize catalog @ 2019-07-15 09:02:23, for catalog: DBNamesChecker, key: x_eneld_succession_successor_card_join_successor_join_plan, triggered by system_id: service-now.com:

2019-07-15 09:02:23 (675) Default-thread-11 SYSTEM txid=18dfd39bdb22 Cache synchronize catalog @ 2019-07-15 09:02:23, for catalog: DBViewNamesChecker, key: x_eneld_succession_successor_card_join_successor_join_plan, triggered by system_id: service-now.com:

2019-07-15 09:02:23 (675) Default-thread-11 SYSTEM txid=18dfd39bdb22 Cache synchronize catalog @ 2019-07-15 09:02:23, for catalog: DBNamesChecker_KnownMissing, key: x_eneld_succession_successor_card_join_successor_join_plan, triggered by system_id: service-now.com:

Release London and all previous releases

Cause Multiple update sets have been ran immediately prior to the start of impact Update sets are known to trigger potentially impactful cache flushes which cause increases in instance response times especially if the update sets contain Database View This is due to a specific known PRB . This issue is encountered specifically when flushing caches required to support database views A single update set can trigger multiple cache flushes during its processing which cause all active nodes to do the same The result is that each application node has to rebuild these caches which is done by querying the database A cache is only rebuilt when a specific transaction that needs to access it and finds it needed to be update For specific caches this typically happens when over a period of several minutes as the various users access specific features However database view data is stored in a common cache that is required by all transactions so requests to rebuild this cache will trigger immediately across all nodes The severity of PRB is directly related to the total number of application nodes in the instance so sub production instances are unlikely to see similar levels of impact, or even be impacted at all Once the cache for database views is flushed all application node immediately start to query the database to repopulate it as soon as the next transaction is triggered If an update set flushes the cache multiple times then this causes multiple queries all running to keep rebuilding the cache causing database connection that leads to performance issues as seen in this impact Resolution Temporary Relief Measure

Restart the most heavily impacted application nodes This terminates all impactful transactions and allows caches to be rebuilt following the restart

Solution

Please upgrade to London Patch 8 Hot Fix 4 or later as the PRB is fixed in this release.

Preventative Measure(s)

In the mean time ServiceNow recommends that if you need to commit any update sets containing updates set you should look to schedule those outside of core business hours.