Instance Observer Best Practices GuideIssue <!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } To interpret the data provided by Instance Observer, it's important to have a basic understanding of ServiceNow architecture. Let's take a brief walkthrough of the main architectural concepts: 1. What are Nodes? Each ServiceNow instance consists of a number of nodes. Think of each node as a separate server carrying out instance processing. This allows you to add more nodes to increase the processing capacity of the instance. For redundancy, there is a set of primary nodes and a set of standby nodes ready to take over in the event of a failure. For example, your instance could have 16 primary nodes and 16 standby nodes. Each node can be one of three types: Generic, which carries out all types of processing; UI, which only processes user transactions; and Worker, which executes ServiceNow scheduled processing jobs and incoming API transactions. Generic nodes are the default, but these can be split into UI and Worker nodes. 2. What are Servlets? Each node has a web server for handling user transactions. This includes a servlet, which is responsible for querying and updating the instance database. This servlet encapsulates the Glide API, providing an abstracted interface to the instance database. There is only one servlet per node, so any servlet metric is also a node metric. 3. What are User transactions? These are transactions initiated from the ServiceNow UI that are processed by the web server and servlet. Each node supports multithreaded transaction processing, allowing transactions to be processed in parallel. For instance, a node may have 16 threads devoted to user transaction processing. There's also a semaphore that shows how many threads are currently actively processing transactions. This is known as the Default semaphore. For example, if the Default semaphore value is 15/16, this indicates that 15 of 16 threads are currently in use in the node. 4. What are API transactions? These are incoming transactions initiated by external systems via the ServiceNow REST API. These run on different threads than user transactions. For example, a node may have 4 threads dedicated to processing API transactions. There's also an API_INT semaphore that shows how many of these threads are in use. For instance, if the semaphore value is 4/4, then all of the API transactions threads are currently in use. 5.What are Jobs? These are tasks that aren't directly related to transaction processing—for example, carrying out end-of-month processing or generating scheduled reports. There are two types of jobs. The first of these is a scheduled job, which runs on a regular schedule. The other type of job is an ASYNC job, which is initiated by a transaction. For example, a user transaction may not need a database update to complete before returning to the user. In this case, the transaction can kickoff an ASYNC job to do the update, which means that the transaction itself completes faster, providing a more responsive experience for the user. Again, there are dedicated threads for job processing. 6.What is Heap? This is the memory in a node dedicated to object and data storage. This is a shared resource across all threads in the node, although each node has its own heap. For example, if a node has a heap value of 1.9/2.2 GB, this means that the heap size is 2.2 GB and 1.9 GB are currently in use. Because the heap is a shared resource, processing in one thread can affect processing in another thread. For instance, if a thread devoted to job processing is using too much heap memory, this can slow down user transactions if there isn't enough heap memory left for transaction processing. Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All Resolution<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Now, let's take a look at best practices for using Instance Observer to manage instance performance. These fall into two categories: • Triage activities. These are diagnostic and remediation activities you need to perform when you experience a performance issue. Typically, these are done using the Triage module in Instance Observer, but you also may need to look at additional data in Instance Observer when diagnosing issues. Note that the Triage module is available for Impact Advanced and Total customers. • Scheduled activities. These are activities you need to perform on a daily, weekly, or monthly basis to maintain instance performance and prevent future issues. Please note that the best practices provided in this document address common instance performance scenarios, but are not intended to provide a comprehensive walk-through of all of the capabilities and metrics available in Instance Observer. For a detailed overview of these capabilities, please refer to the Instance Observer product documentation. You can also find detailed information about the Triage module in KB0010061 in Instance Observer. Triage activities Triage activities let you diagnose and remediate performance issues. You should perform these whenever you have a performance problem. Here, we look at three common types of performance issue: User transaction performance issues - These are performance issues detected at the ServiceNow UI. Examples include slow transactions, a non-responsive instance, or the inability to access an instance API transaction performance issues - These are performance issues with incoming transactions from external systems via the ServiceNow REST API, including slow transactions and an instance that doesn't respond to API requests. Job performance issues. These are performance issues related to jobs running in the ServiceNow instance, such as delayed notifications, slow Flow Designer flows, and other jobs taking an anomalously long time to complete. Getting a Triage snapshot To perform triage, you'll need to get a snapshot of your instance triage data. Do this by clicking on the triage module in Instance Observer, choosing the instance you want to triage, and clicking "Triage" as shown below. Note that you are limited to 10 snapshots per day to avoid excessive loads on your instance. Now, let's take a deeper look at the three types of performance issues we just identified. User transaction performance issues User transactions often experience performance issues because there aren't any threads left to process these transactions. This causes transactions to queue up, delaying execution—or, in severe cases, making the instance non-responsive. Check the Default semaphore The first thing to do is to check the Default semaphore in each node to see how many threads are in use. To do this, click on the "See More" in the Semaphore Sets widget in the Triage module. You'll see a list of nodes with the current values for each semaphore in the node, as shown below. Identify all primary nodes that have a Default semaphore with more than approximately 90% utilization (for example, 14/16 or greater). If you don't know which are your primary nodes, refer to Appendix A(In Related links) Note that this semaphore usage metric is a time-averaged value, so even if the semaphore usage isn't 16/16, the instance may still be running out of user transaction processing threads intermittently, resulting in performance issues. If all of your primary nodes are at 90% utilization or above, raise a case with Now Support right away. This indicates a systemic problem within your instance affecting all nodes, such as a backend database issue. Check the available heap space otherwise, the next thing to do is to check the heap space in the affected nodes. High heap space utilization can result in transaction performance issues. To do this, click on the Servlet widget in the Triage module to see a list of key servlet metrics for each node, as shown below: Identify all of the affected primary nodes with heap space utilization above 90% (for example, 2.0/2.2 GB or above. This indicates a potential heap space issue in these nodes, but you still need to carry out further diagnostics to confirm this. Why? Because high heap space utilization can be completely normal —heap space utilization moves up and down as the node carries out garbage collection activities on the heap. To determine if heap space utilization is actually a problem, you need to look for anomalous behavior of the heap space over time. To do this, navigate to Performance > Node Health in Instance Observer and click on "Get Snapshot". Look at three metrics: Node Garbage Collection Time, Node Memory Max, and Node Memory Mean. The screenshots below show normal and anomalous behavior for each of these metrics. Note that these graphs by default show data for all of the nodes in the instance, but you can see data for a single node by rightclicking on the corresponding node name below the graph. If you observe anomalous behavior for any of these metrics, high heap space utilization is likely the cause of the user transaction performance issues. If you see this, it may be due to long-running transactions or jobs. However, if, after investigation (see below), you determine neither of these are responsible for the anomalous heap space behavior, raise a case with Now Support. Check long-running transaction One key reason for running out of user transaction processing threads (as indicated by the Default semaphore) is long-running transactions. In some cases, these can also lead to high heap space utilization. To check long-running transactions, click on "View Details" in the Transactions widget in the Triage module to see a list of transactions across all nodes, as shown below: If you observe anomalous behavior for any of these metrics, high heap space utilization is likely the cause of the user transaction performance issues. If you see this, it may be due to long-running transactions or jobs. However, if, after investigation (see below), you determine neither of these are responsible for the anomalous heap space behavior, raise a case with Now Support. Check long-running transaction One key reason for running out of user transaction processing threads (as indicated by the Default semaphore) is long-running transactions. In some cases, these can also lead to high heap space utilization. To check long-running transactions, click on "View Details" in the Transactions widget in the Triage module to see a list of transactions across all nodes Look for long-running jobs (for example, more than 30 minutes) on the nodes you previously identified as having anomalous high heap space utilization. Also identify any patterns, such as many ASYNC jobs with the same name. Take note of the Node, Job Name, and Started value for these jobs. In your instance, in the left-hand navigator bar, navigate to "Transaction (Background)" to see if any of these jobs are significant contributors to overall job execution times. If there are long running jobs or many similar jobs, these may be responsible for the heap space problem. Kill these jobs, since this may provide short-term relief. You can find instructions on how to kill a job here. Also, if a job is running because of an ASYNC Business Rule (jobs with the prefix ASYNC), then temporarily disable the Business Rule to prevent a further influx of jobs. Then, review the jobs to identify and fix any issues. You can find information on how to troubleshoot jobs here. If you're unable to successfully troubleshoot a long-running job, raise a case with Now Support API transaction performance issues Triaging API transaction performance issues is similar to triaging user transaction performance issues: • Start by looking for primary nodes that have an API_INT semaphore with high utilization (for example,4/4). You can find this information by looking in the Semaphore widget in the Triage module. If all of your primary nodes show high utilization, then raise a case with Now Support • Check the heap space for the affected nodes. This is the same process as described previously for user transaction performance issues. • Look for long-running API transactions. Follow the process described previously for long-running user transactions, but in this case look for transaction URLs that start with "/api". Most URLs that start with "/api" are integration transactions.Troubleshoot any API transactions that have issues. You can find information about to troubleshoot inbound integrations in KB0564204. • Also look for long-running jobs or large numbers of jobs with similar names. This is the same process described previously for user transaction performance issues. Note that if jobs are causing issues by creating heap space problems or other backend issues such as excessive database access, you are also likely to see user transaction performance issues. • If after following all of these steps, you are still experiencing API transaction performance issues, raise a case with Now Support job performance issues In this case, there are delays in job processing. Jobs are processed on their own worker threads, but, unlike user and API transactions, there isn't a corresponding semaphore that shows you how many threads are busy. However, you can directly check this using the Scheduler widget in the Triage module. Click on "View Details" in the widget, and you'll see scheduler information for all of the nodes in your instance, as shown below: will Look at the Workers column to identify primary nodes with a high number of busy workers (typically 7/8 or 8/8). For each of the affected nodes, look at the jobs on the node. You can do this using the Jobs widget in the Triage module. This is the same process described earlier for checking jobs when triaging user transactions. Identify long-running jobs and large numbers of similar jobs. For example, one common scenario is a flood of ASYNC jobs overloading worker threads due to incorrect or recursive code. If you do see a flood of ASYNC jobs, you may want to temporarily disable the corresponding business rule to prevent further flooding. You may also want to kill long-running scheduled jobs to provide temporary relief. If there are a significant number long-running scheduled jobs or a flood of ASYNC jobs, review the jobs to identify and fix any issues. You can find information on how to troubleshoot jobs here. If you are experiencing job delays but there are available workers and few long-running jobs, this may be due to anomalously high heap space usage slowing down job execution. Follow the same process previously described in this document to identify anomalously high heap space usage. Then investigate potential causes of this, such as long-running user or API transactions, as described previously. If you're unable to resolve job performance issues after following these steps, raise a case with NowSupport. Monitor self-service alerts If you are an Impact Advanced or Total customer, you can configure custom alerts. These are alerts created by applying thresholds to metrics collected by Instance Observer. You should check your alerts every day to determine whether any abnormal conditions have occurred and take appropriate action. Configuring alerts Instance Observer provides a large number of out-of-the-box alerts that you can enable and configure. However, you should only enable the subset of alerts you need to proactively detect abnormal conditions in your ServiceNow instance. To help with this, Instance Observer uses artificial intelligence to recommend alerts you should enable based on the behavior of your instance. To configure alerts, navigate to Alerts > Configure Alerts. You'll see a list of available alerts, as shown below. The list shows you which alerts are enabled and which aren't, and it also shows you which alerts are recommended. To enable and configure an alert, click on the alert and you will get a fly-in screen that leads you through the configuration process. As part of this process, you will be able to set thresholds and simulate the number of alerts that would be generated based on these thresholds. This allows you to fine-tune your thresholds so you don't receive an unnecessarily high number of alerts. You can also set a notification for the alert, which sends a message to a configured email address when the alert is triggered. Monitoring alerts To see the alerts that have been triggered, navigate to Alerts > Alert Console. This shows you all of the alerts for the last seven days, as shown below. You can also click on alert to see a chart of the underlying metric data that led to the alert being triggered. You can also download the alerts as a CSV for future reference. Note that if you are seeing an excessive number of alerts of a specific type, you may want to reconfigure the thresholds for that alert. Review daily performance To identify current performance issues that may be affecting your instance, review the Daily Performance Metrics Report. To do this, navigate to Reports > On Demand. Then, set the Report Type drop-down to "Daily Performance Metrics Report", selecting the instance you want data for in the Instance drop-down. Click "Generate Report". You'll see a report similar to the one shown below. To understand the content of this report, you can find information about individual metrics in Instance Observer help. Navigate to Help > Metrics Overview and click on the metric in the Article Topics list. Look at the Top 10 Slow Transactions table in the report and review them to identify excessive transaction times. Focus on user transactions, since these may be affecting your end users. To identify user transactions, look for transactions with a URL that doesn't start with "/api". Troubleshoot these transactions using the same process described previously for triage. Also look at the top 10 Long Running Jobs table. Troubleshoot these jobs using the same process described previously under "Job Performance Issues". Finally, review the Critical Support Cases table. If there a case with a state "Work in progress – Awaiting info", review the case in Now Support and provide the requested information. Weekly activities Review scheduled jobs On a weekly basis, you should review details about jobs that are running in your instance, as well as information about overall scheduler performance. To review job details, navigate to Performance > Job Details in Instance Observer. Select "Last 7 days" in the "Date Range" drop-down and "Select All" in the "Metrics" drop-down. Then, click "Generate Snapshot" to get current job details, as shown below. Look at the top graph. This shows hourly jobs and transaction counts over one day. Scroll back through the last seven days using the scroll tool above the graph. You should see a consistent pattern across all seven days. If there is an obvious change to the pattern, click on the affected bar(s) and select "View details" for a breakdown of the jobs that were executing during that hour time interval. Look for jobs with high job counts, and compare these job counts to the same hourly job counts in previous or subsequent days. Identify any anomalies and investigate these jobs to see why there was a change. Also, if you are an Impact Total customer, you can find additional anomalies by clicking on the "Anomalies" tab above the graph. This will show a graph of anomalies across all of the time periods. Click on any anomalies bar and click "View details" to see anomalous jobs. Also look at the Top 2K Executing Jobs table (one of several tables below the graph). Look at the current duration of these jobs to see if the duration is longer than the scheduling period for the job. For example, is an hourly job taking more than an hour to run? If so, investigate the job to see why execution is taking so long. Also, look at the Overdue Jobs in the Top 2K Pending Jobs table. An overdue job means that the scheduled start time of the job has passed but the job has not started executing. This can indicate overloaded Scheduled Job processing. Use the Triage module as described in the section "Job performance issues" to see if all the scheduler threads are busy and if so, what jobs are making them busy. Follow the instructions in that section to troubleshoot the job performance. Next, review scheduler metrics. To do this, navigate to Performance > Scheduler in Instance Observer. Select "Last 7 days" in the "Date Range" drop-down and "Select All" in the "Metrics" drop-down. Then, click "Generate Snapshot" to get scheduler metrics, as shown below. This page contains three graphs: Completed Scheduled Jobs, Scheduler Mean Queue Age, and Scheduler Mean Queue Length. Look for spikes in the Scheduler Mean Queue Age that lasted more than 30 minutes, since this can indicate that a node is struggling to process jobs in a timely manner. You can find troubleshooting instructions for this in Instance Observer help. Navigate to Help > Metrics Overview and click on Scheduler Mean Queue Age in the Article Topics list. Also look for spikes in the Scheduler Mean Queue Length, since this can also indicate that a node is struggling to process jobs in a timely manner. You can find troubleshooting instructions for this in Instance Observer help as well by clicking on Scheduler Mean Queue Length in the Article Topics list. Trend top transactions You should also trend top transactions every week to identify transactions that are running more slowly than before or are running more frequently. Before you trend top transactions for the first time, watch the tutorial for top transactions video on the Instance Observer Help page to familiarize yourself with the Top Transactions tool. To view top transactions, navigate to Analytics > Top Transactions, set the Date Range drop-down to "Last seven days", and click on "Get Snapshot". Once the snapshot appears, click on the Detailed tab to see a list of top transactions, as shown below. By default, this list is ordered by total execution time, which is the total time spent executing the transaction in the last seven days—for instance, if the transaction ran 10,000 times, then this is the aggregate time across all 10,000 executions. Clicking on a transaction displays graphs of the transaction count and average execution time. You can also see metrics at the top that show how these counts and execution times have increased or decreased since the previous seven-day interval. As a rule of thumb, just focus on the top 20 transactions based on total execution time, since these will have the most impact on your instance performance. Look for transactions that have become slower this week. Ask yourself if this is still an acceptable execution time given the nature of the transaction, or whether you need to investigate why it is running more slowly. Also look for significant increases in transaction counts and determine why these are taking place—for example, were these anticipated because you added new users to the instance, or were they unexpected. Once you have identified problematic transactions, you need to troubleshoot them. You can do this using the instructions in KB0997495. As part of this troubleshooting, you may have to use the Node Log File Browser. You can find documentation for the browser here, and further instructions on using KB0584420. If you're unable to troubleshoot a transaction, raise a case with Now Support. Finally, note that certain types of transactions are not included in transaction time metrics. These include AJAX and AMB (Asynchronous Message Bus) transactions, which means that Performance Analytics dashboards, reference field pop-ups, and some Workspace Experience, Portal Experience, and Service Portal features are not covered. In addition, the execution time metrics are for actual server-side execution and do not cover network or browser delays. Monthly activities Monitor table growth rates Every month, monitor your database tables for excessive growth. To do this, navigate to Reports > OnDemand. Then, set the Report Type drop-down to "Table Growth Report" and set the Date Range to "Last 30 days", remembering to select the instance you want data for in the Instance drop-down. Click "Generate Report". You'll see a report similar to the one shown below. First, look at your overall database size. Has it grown more than 10% in the last month? If so, are there any obvious reasons for this, such as importing significant amounts of external data using Import Sets? If so, this database growth may be normal. If databases growth is abnormal see "KB1518213 Mastering Data Management in ServiceNow: from tracking growth to efficient cleanup". Next, look at your top 20 tables by size (see the table in the report). If the syslog tables appear in the top 20 and have grown significantly in size compared to the previous 30 days, check that you have not left any debug settings on after previously troubleshooting an issue, since this will cause these tables to grow. The syslog tables are a series of separate physical tables that are rotated. See "Demystifying table rotation, extension, and table cleaner". If any other table in the top 20 has grown significantly (except for sys_attachment_doc), investigate why it has grown. For example, if sys_audit has grown unexpectedly, review recent auditing changes and turn off auditing of some tables and fields if appropriate. Review slow query patterns Review slow query patterns every month. Slow queries are database queries that take longer than normal to execute, often due to query design or other issues such as lack of indexing. Before you review slow query patterns for the first time, watch the Tutorial for Query Patterns video on the Instance Observer Help page to familiarize yourself with the Query Patterns tool. To review slow queries, navigate to Analytics > Query Patterns, set the Date Range drop-down to "Last 2 Weeks", and click "Get Snapshot". Then, click on the Details tab to see slow query details as shown below. By default, this list is ordered by total execution time, which is the total time spent executing the query in the last two weeks —for instance, if the query ran 10,000 times, then this is the aggregate time across all 10,000 executions. Clicking on a query displays graphs of the query count and average execution time. You can also see metrics at the top that show how these counts and execution times have increased or decreased since the previous two-week interval. As a rule of thumb, just focus on the top 20 queries based on total execution time, since these will have the most impact on your instance performance. Look for problematic queries. For instance, a 10-second query is problematic if it is executed once a minute or faster, but it's not problematic if it's only executed once an hour. Similarly, a 100- millisecond query is problematic if it's executed every second or faster, but it's not problematic if it's executed every 10 seconds. Once you have identified problematic queries, you can find information on troubleshooting them here, here, and here. Once you have completed reviewing slow queries for the last 2 weeks, review slow queries for the previous two weeks. You can do this by setting the Date Range drop-down to "Custom Range" and setting the appropriate range. Trend top transactions This is the same process as described previously under weekly activities. However, instead of looking at 7 days, this looks at an entire month to identify and address longer-term trends. Do this by setting the Date Range to "Last 2 weeks" and analyzing the top transactions as before. Then, set the Date Range drop-down to "Custom Range" to analyze the previous two weeks. Keep in mind that what may be an acceptable seven-day trend may not be acceptable if it continues over a month. That's why it's important to look at longer-term trends. For example, if a transaction becomes 2% slower over seven days, that may be acceptable. However, if the trend continues over an entire month, and the transaction becomes 8% slower in total, then you may need to take action depending on the business impact. Related Links<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Appendix A: identifying primary nodes To identify the primary nodes in your instance, navigate to Performance > Node Health in Instance Observer and click on "Get Snapshot". The primary nodes will appear under each of the three graphs on this page, as shown below.