List of Checks for Instance Troubleshooter - Support and Troubleshooting

List of Checks for Instance Troubleshooter Summary PLEASE NOTE:

Beginning with the Now Platform® Xanadu release planned for September 2024, we will no longer deploy or activate the application listed below to new instances. It will remain active in your instance, including when you upgrade to a new release family. Beginning with the Now Platform® Yokohama release planned for March 2025, we will end support for the application listed below. While it will remain active in your instance, including when you upgrade to a new release family, we will not be able to support it. Instance Troubleshooter, a free store application, helps ServiceNow instance administrators to resolve issues on their instance by themselves. This application can detect issues in the instance across several product categories. When the administrator invokes the troubleshooter, each product troubleshooter runs one or more checks and reports its findings with detailed information and links to resolving the issues by themselves. This KB article does not go into detail about this application. Please view KB0870978 - Instance Troubleshooter User Guide to get a full description of the application.

This KB article lists all the checks in versions 1 to 3.1 of the Instance Troubleshooter application. Note that the checks introduced in version 3.1 are marked with (new) .

Categories Access Control Authentication Clone Core Platform Currency Discovery Email Forms and Fields Import/Export MID Server Mobile Performance Performance Analytics Reporting Service Catalog Service Portal SLA User Experience Workflow

Access Control Priority Short Description Description 2 - High

ACLs without roles or condition or scripts may cause unintended results

ACLs without roles or condition or scripts may allow unintended users to update/delete/create/read records.

2 - High

ACLs using getRowCount may cause performance issues

ACLs using GlideRecord's getRowCount property may cause performance issues when working on tables with a high record count.

2 - High

High Security Settings plugin is disabled on the instance.

The High-security plugin sets more secure defaults. This should be activated. However, due to its impact on the whole instance security/accesses, testing should be completed on a sub-prod instance before enabling your production instance.

Authentication Priority Short Description Description 1 - Critical

SAML 2.0 certificate is either missing the X509 data or invalid

If a user is trying to login and the error "Could not find a digital signature in the inbound SAMLResponse" appears in the system logs, then the certificate must be validated or updated.

2 - High LDAP Authentication check If LDAP Authentication is enabled but the LDAP Servers are using MID Servers for connection, then LDAP Authentication is not supported. If all LDAP Servers on the instance use a MID Server, then LDAP Authentication can be safely disabled. Otherwise, it needs to remain enabled as the LDAP Servers that do not use a MID Server may be used for LDAP Authentication. 2 - High X.509 Cert for SAML Authentication check Multi-Provider SSO Identity Provider records should have Metadata URL populated. A lot of SSO related cases are due to Certificate Mismatch. Having the metadata URL populated will allow the instance to automatically fetch/update the certificate every 30 minutes. If the SSO provider is reachable and has a metadata URL, this should be populated. 2 - High IdP certificate has changed or expired If the IdP certificate has changed or expired the IdP certificate should be updated on the ServiceNow instance to match the one updated on the Identity Provider end. 2 - High Auto Redirect IdP is enabled and SSO is down If instance has 'Auto Redirect IdP' enabled and if Identity Provider is down or having issues, users automatically get redirected to the IdP login page without having an option to login using local DB login. 2 - High Cloning from this instance will overwrite SAML SSO on the target If the Integration - Multiple Provider Single Sign-On Installer [com.snc.integration.sso.multi.installer] plugin is activated on the source, the Identity Provider [saml2_update1_properties] table should be excluded from the clone and the glide.authenticate.sso.redirect.idp should be preserved. Otherwise, users will not be able to login to the target instance after the clone. 2 - High Identify invalid Multi SSO configuration in the instance You can set one IdP as the primary IdP to which new users are automatically redirected when they access the base instance URL. You can also set one IdP as the default IdP.

Checking if the property (glide.authenticate.sso.redirect.idp) is linked to a valid identity provider record. 3 - Moderate (new) New LDAP Server Record Producer is missing New LDAP Server Record Producer is missing. This record producer is required in order to create new LDAP Server records. 4 - Low Checks if CMS is properly configured for SSO This checks if cms pages are configured for SSO. By default, cms pages are public and will not use SSO. If SSO is desired for cms pages, please apply the suggested resolution. Clone Priority Short Description Description 3 - Moderate

Check if clone target set to true for prod instance.

The property glide.db.clone.allow_clone_target for production instances is default set to false, but in some scenarios it can be set to true to clone over the production instance but then not reverted after the clone. We have detected that this instance property is set to true.

3 - Moderate

Cannot exclude database views

Database views are not physical tables, they are virtual tables built from connecting other physical tables.

Therefore to exclude data you need to exclude from those tables, not the database view.

3 - Moderate

Clone: Check for invalid/Incorrect Clone Targets

Running a simple API check to validate if the clone target returns a response or not. If we do not receive a response this is flagged as a potential invalid clone target.

3 - Moderate

Tables to not exclude in clone

Checks for specific tables that should not be excluded in the clone engine.

Core Platform Priority Short Description Description 1 - Critical

JSON script include should be functional

JSON script include should exist and be functional. Some of fundamental features including clone do not work without it.

1 - Critical Escape HTML should be enabled A malicious user can inject HTML code within the form field to execute unwanted scripts on different client/user sessions if the property "glide.ui.escape_html_list_field" to false 1 - Critical Restrict unauthenticated access to images Use the glide.image_provider.security_enabled property to control the security settings for images. If set to true, images are visible only to authenticated and authorized users. If set to false, images are visible to anyone with a URL to the attachment. 1 - Critical Change mangagement Cancel UI action doesn't work After upgrading to Quebec Patch 0 or 1, the Cancel UI action doesn't work if the "Change Management Best Practice - Jakarta" is not installed 2 - High Check if text index is running on a table or hasn't been run. Checking all the text indexes on instance to detect which ones have not been indexed or currently being indexed. 2 - High Language picker is not showing options to select Even when at least one language plugin is active, the language picker does not show the choices. 2 - High Attachment name should not contain '&' If attachment name contains any special character like '&', after attaching it converts to &amp. This can cause issues if we are searching by file name 2 - High Checks for PRB1437770 This check will determine if sys_security_acl.do?sys_id=90029ca10a0a0b440114ffbb79bfe100 contains the following roles: report_scheduler, report_global, report_publisher, report_group as per PRB1437770. 2 - High Enabling debug properties may cause performance issues. Debug properties should be enabled when performing system testing in the instance, post that these properties should be set to false. Enabling these properties could unnecessarily bloat the size of the database and cause disk size issues. 2 - High Background script to identify fields which are being superfluously text indexed The ServiceNow platform uses the Zing engine to perform text indexing and text searching of data held in various tables and fields within an instance. Out of box the platform text indexes a number of fields which are generally not useful for text searching (as users are unlikely to ever search for the type of values they contain). In large instances this can cause:

* A large number of text_index events to be created * Underlying text index tables to grow extremely large * Degradation to text search performance * Degradation to text index rebuild performance (should a text index need to be regenerated - for example following a clone)

A much better approach is to:

* Define the types of fields which are generally superfluous to text index either because they contain data which users are unlikely to text search against (i.e. duration) or simply contain a small set of repeated values (i.e. sys_class_name) * Walk all tables in a given table hierarchy looking for fields which are one of the above types and where text indexing is currently enabled * Corresponding fields can then have text indexing disabled (if applicable) without needing to go through repeat iterations of reviewing text_index records in the sysevent table 2 - High Performance issues while accessing User records If last_login. and last_login_time fields are audited, then the activity on the user record increases over time which will result in slowness when accessing the user record 2 - High Users noticing page refresh User preference home_refresh defines the number of seconds after which the page is forced to reload for a User. 2 - High No User has Security_admin role Check if there is at least one security admin in the instance 2 - High Longer session time out, may cause performance issues The default system user session timeout is 30 minutes, increasing this value could cause performance issues. 2 - High Unsupported globals and API in Portal client scripts Using certain globals and APIs in Client scripts is not compatible with Service Portal / Mobile. Using these APIs won't work as expected as they do in Platform UI. 2 - High Check to ensure glide.ui.escape_text is set to true The recommended value for the system property glide.ui.escape_text is: true. This property will escape xml and ensure that browsers do not parse or execute potentially malicious javascript embedded in untrusted data. 2 - High Check if default admin is inactive in instance Many jobs/events/processors in the instance have Run as default System administrator which might fail if the admin is inactive/locked out 2 - High Large attachment size, May cause performance issues The default maximum size limit for an attachment file that can be added to a record in the ServiceNow platform is 1024 MB for an out-of-box instance. Large attachments can sometimes cause performance issues on the system and unnecessarily bloat the size of the database. 2 - High Check for invalid roles, groups and inheritance Scan through user-related tables to find any records that are invalid and broken.

EG: Empty roles assigned to groups Empty roles assigned to a user Roles assigned to an invalid user. 3 - Moderate Using operator (^NQ) in encoded queries causes incorrect reference links in the When using the operator ^NQ in an encoded query within a business rule, the results appear in the list view, but the reference links are not correct.

A filter on a business rule that has a query with the operator ^NQ (second level condition) is applied only on the first part of the query. Although the business rule works correctly in the list, if a filter like Active=true is added, the filter is applied only on the first part of the query (before the ^NQ). 3 - Moderate Unable to edit sysauto_script in Quebec Happens because the following ACL on the table level has the admin overrides unchecked. 3 - Moderate Enforce strict user image upload Use the glide.security.strict.user_image_upload property to enable Access Control for the upload/update of a profile picture when performed on a user record.

This setting opens the possibility of an unauthorized user uploading an image to another user's profile. When you set this property to false, an authenticated user could upload an image to another user's account without authorization. 3 - Moderate Check to ensure the system timezone is not inactive on the instance If the system timezone is inactive, users may be unable to select a timezone in the general system settings. Alternatively, duplicate timezones may be displayed for some users (PRB1448481). 3 - Moderate Database debugging on - may cause performance issues. Having this property on causes many logs to be printed in our localhost logs - it may cause performacne issue. We strong recommend setting this off. 3 - Moderate Check if property glide.set_x_frame_options is set to the recommend value: true ServiceNow recommends setting the system property glide.set_x_frame_options to true. This will prevent content from being embedded in iframes and protect against clickjacking attacks. While the recommended value is true, do note that if you're using a 3rd party integration that renders your ServiceNow content in an iframe, you may wish to set the value to false. 3 - Moderate Important deleter jobs should be enabled Key deleter jobs should be enabled like 'Table Cleaner' and 'Import Set Deleter' should be enabled to ensure the relevant tables are kept as trimmed as necessary. Should these jobs remain disabled for a long time, the tables will grow to be very large and transaction performance will decrease over time. This can also lead to disk space capacity issues as the instance is not cleaning up the temporary/transient records. 3 - Moderate Check for activity stream on sys_user table This check will determine if the activity stream is added to the sys_user form. This can cause user records to load slowly. 3 - Moderate Client Scripts defined on the Global table A global client script is any client script where the selected table is Global. Global client scripts have no table restrictions; therefore they will load on every page in the system introducing browser load delay in the process. There is no benefit to loading this kind of scripts on every page 3 - Moderate (new) Verify there are no Orphan records for Connection [sys_connection] table Verify there are no Orphan records for Connection [sys_connection] table.

Navigating to sys_connection table shows a list of records. However, any attempts to open up the records fail with the error 'Target Record Not Found'.

This issue may happen after a clone that that was done recently.

4 - Low Rollback option is available on fix script form. The Fix scripts module is very handy to run scripts on your instance. One thing that is hidden in the record for rollback functionality, out of the box the rollback option is not on the form. If on the form you can check the box and make it so you can record the fix script in the rollback engine and roll back completely if any issues. 4 - Low (new) Check if "Clean Expired Rollback Contexts" is active. If rollback are not cleaned up, shadow tables can grow extremely big, it would cause degrated performance when deleting records from the tables and longer altering time. Currency Priority Short Description Description 4 - Low

Multi currency mode enabled validate records are using correct currency

Multi-currency mode enabled validate records are using the correct currency

4 - Low

Validate exchange rates exist for active currencies

Validate exchange rates exist for active currencies. This does not cover the fx_currency type.

4 - Low (new)

Single currency mode - validate currency records match locale

Single currency mode is enabled. Checking to validate records are using the correct currency

4 - Low (new)

Single currency mode - validate price records match locale

Single currency mode enabled validate records are using correct currency

Discovery Priority Short Description Description 2 - High

Table for credential class should exist

Discovery will fail if discovery_credentials.sys_class_name does not have a value, or the corresponding table does not exist.

2 - High

Number of active credentials per class should not be more than 20

Number of active credentials per class should not be more than 20. The more active credentials there are per class, the more time it will take for discovery to scan.

2 - High

sys_mod_count should not be more than 1000 for Credentials

Frequent credential updates will lead to more frequent cache refreshes on MID Server. Frequent cache refreshes will increase the overall overhead to the Instance and MID Servers.

3 - Moderate (new)

CAPI to Patterns Migration check

Please refer to the below knowledge article to perform CAPI to Patterns Migration https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0827153

3 - Moderate (new)

Custom Business Rule on cmdb tables with abort action code.

Custom Business Rules with abort action code on CMDB tables causes discovery issues.

3 - Moderate (new)

Check CMDB containment rules are out of box

Customized containment rules can cause issues in Discovery so this check would help you to identify this customization to take corrective actions

3 - Moderate (new)

Check CMDB deletion strategies are out of box

Customized deletion strategies can cause issues in Discovery so this check would help you to identify this customization to take corrective actions

3 - Moderate (new)

Check CMDB hosting rules are out of box

Customized hosting rules can cause issues in Discovery so this check would help you to identify this customization to take corrective actions

3 - Moderate (new)

Check CMDB relationship types are out of box

Customized relationship types can cause issues in Discovery so this check would help you to identify this customization to take corrective actions

3 - Moderate (new)

Custom Mandatory CMDB attributes

Custom Mandatory CMDB attributes cause IRE to fail with required attribute missing errors. Review these custom mandatory attributes to avoid IRE issues.

3 - Moderate (new)

Check Pre & Post Sensor script are out of box for Discovery

Customized Pre and Post scripts can cause issues in Discovery so this check would help you to identify this customization to take corrective actions

Email Priority Short Description Description 1 - Critical

Inbound email disabled

If the glide.email.read.active property is set to false, then the instance is currently configured to NOT receive any email

1 - Critical

Ensure the OOB POP3/SMTP accounts match the instance name

Customers may have accidentally cloned the POP3/SMTP accounts from other instances, causing emails to be read by the wrong instance.

1 - Critical

Guest account does not exist

The out-of-the-box guest account is necessary when processing some inbound actions. For example, when the system needs to create or update records and the sender is not from a valid domain.

1 - Critical

There is only one active SMTP account

There is only one active SMTP account.

1 - Critical

Email Reader job not running

The EmailReader job is responsible for retrieving email messages from the POP3/IMAP server. If this job is not running, email from this server is not currently being retrieved.

1 - Critical

SMTP Sender job does not exist

The SMTPSender jobs are responsible for sending email messages to the SMTP server. If these jobs are not present, then no email will be sent from the instance.

1 - Critical

SMTP Sender job not running

The SMTPSender jobs are responsible for sending email messages to the SMTP server. If these jobs are not running, then no email will be sent from the instance.

1 - Critical

Email Reader job does not exist

The EmailReader job is responsible for retrieving email messages from the POP3/IMAP server. If this job is not present, email from this server is not currently being retrieved.

1 - Critical

Outbound email disabled

If the glide.email.smtp.active property is set to false, then the instance is currently configured to NOT send any email.

2 - High (New)

Basic Auth as authentication method for Microsoft Email servers is deprecated.

Basic Auth as authentication method for Microsoft Email servers is deprecated. Check if instance has any email accounts connecting to Microsoft using basic auth and notify customers that they should use OAuth.

2 - High (New)

SMTP sender email values must be valid email addresses

SMTP 'Email Account' records' 'From' field [sys_email_account.from] value is used as the email address when sending email from this account. If this value is empty, then the User name will be used as the return address for notifications sent from the instance. It is best practice to include a valid email in the 'From' field. This check is marked as 2-High because email sending from this account will fail if:

* The 'From' field [sys_email_account.from] contains an invalid email address

* The 'From' field [sys_email_account.from] is empty an invalid email address is used as the User name [sys_email_account.user_name]

2 - High

Guest account is not active

The out-of-the-box guest account is necessary when processing some inbound actions. For example, when the system needs to create or update records and the sender is not a valid user. A guest account is present in the instance, but it is inactive or locked out. This is not necessarily an error, but it just means inbound emails from unknown users will be marked as received-ignored.

2 - High

SMTP Sender stuck

The SMTPSender jobs are responsible for sending email messages to the SMTP server. This job runs once every 2 minutes. If the SMTPSender job has not been updated in a few minutes, then it may be stuck causing outbound email to be delayed.

2 - High

Email Reader stuck

The EmailReader job is responsible for retrieving email messages from the POP3/IMAP server. This job runs once every 2 minutes. If the EmailReader job has not been updated in a few minutes, then it may be stuck causing inbound email to be delayed.

2 - High

Securing Inbound actions

Verify all the senders must have required role to trigger inbound action.

3 - Moderate

All outbound email is directed to one address for debugging purposes

All outbound email is sent to one email address. This is not necessarily an issue if an instance is not live and debugging activities are being performed. However, if email is not being currently received by your users, then one possibility is that somebody left this debug property enabled.

3 - Moderate

Outbound email with large recipient list

An outbound email with a large recipient list may slow down email sending.

3 - Moderate (new)

TD Tags in Email Client Template adds unwanted extra lines

This is caused by PRB1547429

Extra spaces get added in the HTML table if there are tags in the email client body.

This will not break any functionality, just causing extra new lines appear inside the table.

Forms and Fields Priority Short Description Description 2 - High

Slowness when opening forms with high number of activities

glide.history.max_entries property specifies the maximum number of entries retrieved for display in an activity stream. The default number is 250.

2 - High

Unable to delete a record using Delete UI Action

When trying to delete a record using Delete UI Action, nothing happens and a console error is thrown : Error: Uncaught TypeError: Cannot read property 'split' of null at getCascadeDelTablesDoneForm

2 - High

HTML fields don't show all actions/options after upgrade

When opening any form with HTML fields, all the available options don't show to insert images/links

2 - High

@mention functionality not working for Journal input type of fields on the form

Users should be able to @ mention other users if the input type of the field is Journal Input type of field. However this doesn't work as expected if live forms is disabled

3 - Moderate

Content from HTML type fields is displays with HTMLtags in activity stream.

HTML Characters appear in the Activity stream for the HTML type fields because the property glide.ui.security.allow_codetag is set to false

3 - Moderate

Check sys_ui_form_section records for invalid sys_ui_section references

This check will find sys_ui_form_section records that contain empty/invalid sys_ui_section references.

3 - Moderate

Items per page has options over 100

Items per page has options over 100,may have performance issues in loading lists

3 - Moderate

Priority data lookup rules are not working on problem table.

Priority data lookup rules does not work on the problem table. When the Impact and Urgency are changed, Priority is not updated. It works fine on the incident table but not on the problem table.

3 - Moderate

Reference click through is not available for read only fields

Before Paris release, even if glide.ui.reference.readonly.clickthrough system property is false, reference icon was showing up. This is actually bug in our system which is fixed from the Paris release as part of PRB1383781

3 - Moderate

Public tags are not visible to everyone

Public tags are not visible to everyone on the forms. The issue is happens if the 'Global' field is set to False in the label_entry record for the particular tag in question

3 - Moderate

Deleting attachment loops forever

When deleting the attachments from the form, the popup hangs and progress keeps on loop forever. The issue is because of the customized UI Page - attachment and UI Macro - attachment_list_body

Import/Export Priority Short Description Description 1 - Critical

ISET Data Source: MID Server must be associated to the Data Source of type LDAP

Data Sources of type LDAP must have a MID Server value set. This MID Server must be up and responding to this instance's requests.

1 - Critical

Data Source record must have a value for the Import Set table name field

The import will not work if the Data Source record does not have a value for the Import Set Table field.

1 - Critical

ISET Data Source: MID Server must be associated to a Data Source of type JDBC

Data Sources of type JDBC must have a MID Server value set. This MID Server must be up and responding to this instance's requests.

1 - Critical

ISET Data Source: Attachment not present

If a Data Source record's type field is set to "File" and File retrieval method is "Attachment", then there must be a file attached to the Data Source record. This attached file has the contents of the data to be imported, therefore the import will fail if this no attachment is present.

2 - High (new)

MID Server used in LDAP Import cannot be part of a cluster

MID Server used in LDAP Import cannot be part of a cluster. A dedicated MID Server should be used for LDAP Data Source imports.

3 - Moderate

ISET record has at least one import set run record completed per transform map

There should be at least one import set run record for each transform map associated to the data source. The import set run record holds information on the transform. If one import set run record exists per transform map, and each of these import set run records's status is set to Completed, then the transform phase was executed and completed. This is marked as 3-Moderate because the transform phase may still be running in which case is not an error but partial transform.

3 - Moderate

ISET record has at least one import set run record per transform map

There should be at least one import set run record for each transform map associated to the data source. The import set run record holds information on the transform. If one import set run record exists per transform map, then at least an attempt was made to transform the data. It is marked as 3-Moderate because it is not an error if these records are not present, it just means that the transform phase was not initiated.

3 - Moderate

At least one record exists in import set table

If at least one record is present in import set table for the given ISET, then the import was good enough for at least one record to be imported. If there are no records for this ISET it does not necessarily mean error, it means that the source just happens to have no data.

3 - Moderate

Cannot "Export to PDF" following upgrade to Quebec

Customer complaining about Export to PDF (Landscape or Portrait) not working after upgrade to Quebec and see a blank screen with below error message. This issue is happening in any form.

3 - Moderate

Check on ImportSetRow to add new columns to the staging table

Instances that contain large numbers of import sets can sometimes become unresponsive when an import adds a column because the instance must alter every row in the staging table. In some cases, the database alter table action causes an outage.

3 - Moderate

Exporting CSV mis aligned due to line breaks

You can control how line breaks appear in exported CSV data using the glide.csv.export.line_break system property.

3 - Moderate

Export results are formatted with the wrong encoding

Export results are formatted with the wrong encoding. Non-English characters (for example, Russian, Hebrew) are replaced by question marks. Cause: The setting for the export encoding is missing or it has the wrong value. This issue is related to exporting data using the CSV format from the instance.

3 - Moderate

When using easy import and an import template dates are not being converted

When using easy import and an import template from table, for example alm_hardware, dates are not being converted correctly. The OOB easy import template, without modifying the import set or transform map, verified the date had been entered correctly in the Excel document. The date looks correct in the import set, but during the transform it is not being updated correctly.

Date in Excel displays 2020-10-10, entered as 10/10/2020, correctly After Import runs, clicking preview data button shows record with Warranty Expiration Date as 10-10-2020 Choosing Complete Data Import transforms the data with a Warranty Expiration Date as 04-11-0016

3 - Moderate

Exporting to PDF can lead to memory issues when default max rows are changed

Sometimes, When exporting reports via list or scheduled reports as pdf, can lead to memory issues.

In many customers change this value as this as to accommodate more columns and rows.

The property to set max rows always plays major role. Glide.pdf.max_rows default is 5000

3 - Moderate

Exporting to PDF can lead to memory issues when default max columns are changed

Sometimes, When exporting reports via list or scheduled reports as pdf, can lead to memory issues. In many customers change this value as this as to accommodate more columns and rows.

The property to setmax columns always plays major role. Glide.pdf.max_columns default is 25.

3 - Moderate

Export CSV limitations check

If doing a CSV export from a list view or using an export set and a column contains more than 32,000 characters, the export will limit those columns to around 32,000 characters.

By default, CSV exports are limited to 32,000 characters per column. The CSV export uses the display value when exporting columns with long string values.

3 - Moderate

Excel Export Cell Limit 2: This check is to verify the allowed number of rows and columns while exporting

Due to the way the platform must load the data into memory while converting to .xls and .xlsx formats, there are known memory issues when exporting large reports to Excel. Therefore, the platform restricts the number of rows/columns you can export. This limitation is controlled via properties:

Row Limits: glide.ui.export.limit, glide.xlsx.export.limit, glide.excel.export.limit Column Limits: glide.xlsx.max_cells, glide.excel.max_cells