Optimizing AI Search incremental indexing performance and addressing AIS Index Event Queue backlog issues that impact the searchability of new or updated recordsIssue <!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } AI Search Index Event Processing can cause performance issues during high load of generated events caused by fields which are being superfluously processed by AI Search. By default AI Search will index all fields on a record in a table that is configured as an Indexed Sources [ais_datasource] record. For each Indexed Source, AI Search incrementally indexes records by generating a [sys_event] record in the ais_index queue. This is triggered by every INSERT, UPDATE, or DELETE action on the record, which is then processed by a single AIS Index Event Processor [sys_trigger] that runs every 30 seconds. This regular processing ensures that any changes to the records are quickly reflected in the AI Search results. Note: The AIS Index Event Processor can take considerably longer than 30 seconds to process events, usually seen during indexing or when the event backlog is growing. The "Queue" field for AI Search single node ingestion is ais_index for AI Search (AIS) indexing events to be processed on the AIS Node(s) containing the indexed data.The "Name" field contains the AIS event name and is ais_index for INSERT, UPDATE or DELETE actions.The "Instance" field contains the sys_id of the record that had an INSERT, UPDATE or DELETE action.The "Table" field contain the name of the table where the record had an INSERT or UPDATE action.The "Parm1" field contains all the fields to be indexed that were INSERTED or UPDATED. With the DELETE action, the "Parm1" field contains the table name where the record was deleted from.The "Parm2" field contains the action. When doing a full table index, it creates the ais_index.no_block event for every 10,000 records to be indexed on the Indexed Source. Once it has indexed 10,000 records, it will set the event is in the state "processed, Scheduled Jobs in [sys_trigger] AIS Index Purge Stale Document and AIS Create Index Updatable Field Event are triggered daily and create an event to be processed at their scheduled times. The ais_index.purge_filtered_doc event is to maintain the Retention Policy settings on the Indexed Source and to remove records from the index that now fall outside the retention policy. The ais_index.updatable_fields event is to support updating dynamically updatable fields in the AIS Index. Once the "ais_index" event is in the state "processed", the index will have been updated on the AIS Node(s) and the AI Search results will reflect these changes made on the record. Symptoms<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } Typically, when a new record is added and is not searchable within 30 seconds, or a record has been updated, but the updates are not reflected in the search results until much later, it can be indicative of an underlying issue with the AIS Index Event Processor. In extreme cases, this processor is taking considerable time to process events, usually caused by a high influx of events generated for processing or a very long running indexing job. This influx of events or a slow indexing job can cause a backlog of events to build up, leading to delays in processing these events in a timely manner. As a result, the search functionality may not be updated in real-time, leading to a discrepancy between the actual data and the search results seen by the end users. Facts<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } There are several tools to monitor the AIS Index Event Queue and Event Processing - 1. AIS Status Page - End of page (Updated every 5 minutes). https://INSTANCE_NAME.service-now.com/xmlstats.do?include=ais 2. AIS Index Queue stats (Updated every 5 minutes) - Add "Created" [sys_created_on] field to the List View using "Sort z to a" on it and review the field Queue Size. https://INSTANCE_NAME.service-now.com/ais_index_queue_stats_list.do 3. System Event [sysevent] table on the ais_index queue showing current backlog of "Ready" and "Queued" ais_index events. https:/INSTANCE_NAME.service-now.com/sysevent_list.do?sysparm_query=queueSTARTSWITHais_index%5EstateNOT%20INprocessed%2Cerror%2Ctransferred 4. Monitor System Events and Scheduled Jobs dashboards Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } All releases. Cause<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } Integrations to third party systems that frequently update ServiceNow tables that are indexed, such as LDAP or SSO, can lead to a significant increase in indexing events. This is because these integrations often update superfluous fields that are not searched on, but do not have the "no_text_index" attribute, and this will create unnecessary indexing events that contribute to a decreased performance of processing index updates. Mass imports or recursive business rules can also contribute to this issue by updating fields that lack this attribute, again generating unnecessary indexing events. Resolution<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } To mitigate this, ensure the "no_text_index" attribute is properly set for fields that do not require indexing, hence reducing unnecessary indexing events leading to better performance when processing index updates. To determine which fields require this attribute, you need to analyze the ais_index queue in the [sysevent] table and group by the "Table" field to determine which table generates the most indexing events. The "Parm1" field contains the fields that were updated on the record to be indexed, however you cannot group by on this field. We recommend that you export the data to CSV or XLS, and do the analysis in a spreadsheet for better visibility. In order to stop it generating the event for the AIS Index Event Processor, all the fields in the event would need to have the "no_text_index" attribute applied to it. The [sysevent] table is rotated, so it only contains events generated in the last 6 days and are then removed after 6 days once processed. Therefore, for example, if the backlog of events were generated by a mass import, you would need to do the analysis within the 6 day timeframe. System Event [sysevent] table on the ais_index queue showing all events generated in the last 6 days, grouped by table. https:/INSTANCE_NAME.service-now.com/sysevent_list.do?sysparm_query=queueSTARTSWITHais_index%5EGROUPBYtablesysevent_list.do?sysparm_query=queueSTARTSWITHais_index%5EGROUPBYtable We also have a script that will add the "no_text_index" attribute on all fields which are being superfluously processed, such as datetime, currency, numerical, etc. fields. Note: We recommend to test either or both of these scripts in a Sub-production instance first. For AI Search [ais_index] use the script in - KB1644352 to identify and disable indexing on fields which are being superfluously processed by AI Search For Zing Search [text_index], use the script in - KB0953781: Background script to identify fields which are being superfluously text indexed Best Practices: In an emergency with heavy flooding, you can stop it generating all AIS Index Update events on a specific table, by making the Indexed Source temporarily "inactive" in table [ais_datasource]. Once the "no_text_index" attribute has been applied to all the required fields, when you make the Indexed Source "active" again, it will need to be reindexed.Review Field settings documentation for "no_text_index" attribute for further information on which fields it can be applied to. Once the attribute has been applied, it will automatically stop generating AIS Index Update events on the field where it was created on.Be cautious with fields like "sys_updated_on" in [task] tables due to retention policies.Re-index the tables after the attributes have been added to remove the field(s) from the index. For tables with more than 10,000 records, we suggest to reindex one table at a time during off-peak hours.Monitor the AIS Index Event Queue ais_index_queue_stats and table [sysevent] where queue="ais_index" and validate the improvements.Apply changes across all instances to ensure consistent performance. Related Links<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } }