Instance Observer alerts - How to troubleshoot Default Semaphore (Mean) alert - DEFAULT_SEMAPHORE_MEANSummary<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } The goal of this article is to provide step-by-step instructions when triaging and troubleshooting Default Semaphore (Mean) Type alert. Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All IO releases Instructions<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Semaphores can be impacted due to multiple reasons. One common issue is Semaphore Exhaustion, which can occur when there is an influx of transactions that quickly deplete the available semaphores. Cache flushes can also impact semaphores, as they can cause a sudden increase in resource usage. Additionally, database impact and Memory issues can contribute to semaphore problems. To narrow down which of these is the contributing factor, follow the below steps. Check for cache flushes, these can cause up to 10-20 minutes of impact. Follow KB0793295 - to check what can cause a full system cache. If it was due to one of the parameters mentioned in the KB, then this should be a problem. Check for Database impact Database impact can be reviewed by going to the Instance Observer -> Performance Tab -> Database. Choose the instance and impact duration. can be reviewed by going to the Instance Observer -> Performance Tab -> Database. Choose the instance and impact duration. Spike in HLL graph could mean that the database is impacted. Due to this, there could be spikes in SQL Response Time as well. Log in to the instance and go to Active Transactions (All Nodes) and kill the longest-running job. Review View and kill active transaction documentation on how to kill the transactions. If the HLL still doesn't reduce, reach out to ServiceNow to review stuck SQL queries. Check for Semaphore Exhaustion. Semaphore Exhaustion causes long wait times as there will be no semaphores left to process incoming user transactions. Login to Instance Observer -> Performance Tab -> Semaphores.As seen in the screenshot, there will be a spike in usage of default semaphores. This means that there could be an influx of transactions. Gathering what might've caused the stuck or long-running transactions is important to find the root cause and stop the influx.To gather stats and threads review KB0517269 - Gathering Stats.do and Threads.do Page Data: On your browser, navigate to https://<instance-name>.service-now.com/stats.do. The <instance-name> should be populated to reflect the instance that is presenting slow response times.Open a new tab or window in your browser and navigate to https://<instance-name>.service-now.com/threads.do.Press Command + s on your Mac or ctrl + s on your Windows PC to save the stats.do and threads.do pages to your desktop. Review the threads for the offending transaction, look for Script Includes and Business rules that might be triggering the transactions. Disable the Script Includes and Business rules causing the influx to provide relief. Click on Transactions from the Semaphore Default Mean graph -> Active Transactions. This will redirect to the instance and open Active Transactions (All Nodes). Review View and kill active transaction documentation on how to kill the transactions. Check Memory issues which can cause node restarts. Go to Performance -> Node health -> Select date range when the impact happened.Check Node Memory Mean – If impacted node is hitting max memory.Check Node Garbage Collection Time – If impacted node has spike occurring.To figure out culprit – Kill the culprit transaction/job Go to Performance – Transaction/Response Times Select the impacted node - Check longest running transactions/Jobs for the impacted node Click on view transactions - Check longest running transactions for the impacted node Click on view Jobs - Check longest running scheduled Jobs for the impacted node If the any of the above conditions happens, open case with servicenow support to capture heapdump and do analysis for memory issue Related Links<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } View and kill active transactionKB0793295 - What can cause a full system cache flushKB0517269 - Gathering Stats.do and Threads.do Page Data