Text search results relevancy score is being capped at 255 - Support and Troubleshooting

Text search results relevancy score is being capped at 255 Issue  Symptoms The customer was executing a text search against the kb_knowledge table but was getting the search results returned in an unexpected order. The customer had configured a number of fields of the kb_knowledge table to have different weightings using the ts_weight field attribute: https://docs.servicenow.com/csh?topicname=t_ControlMatchRelevanceByField.html&version=latest With the specified field weightings, the results were not being returned in the order expected eg. most relevant article first in the results list. Debug of the text search revealed the scoring of each of the records returned by the search. for example: other (not gif, sql)14:15:20.603: >>>> other (not gif, sql)14:15:20.606: 1: Score for document 1631afebdbf91b00afe666a25b9619bd is: 255.0 other (not gif, sql)14:15:20.606: 2: Score for document 876fdbe7dbf91b00afe666a25b96193a is: 255.0 other (not gif, sql)14:15:20.606: 3: Score for document e14fdf2bdbf91b00afe666a25b9619bd is: 255.0 other (not gif, sql)14:15:20.606: 4: Score for document 1f8e93e7dbf91b00afe666a25b961919 is: 255.0 other (not gif, sql)14:15:20.606: 5: Score for document ec7943a3dbb91b00afe666a25b9619e5 is: 255.0 other (not gif, sql)14:15:20.606: 6: Score for document 4c5e17a7dbf91b00afe666a25b96199d is: 252.0 other (not gif, sql)14:15:20.606: 7: Score for document af7ddb27dbf91b00afe666a25b961969 is: 250.0 other (not gif, sql)14:15:20.606: >>>> The scores were being capped at a maximum value of 255 and thus the most relevant record were not being displayed at the top of the list. The results were effectively random as they all had the same score value. It was expected that the scoring of the records would have followed the process documented here: https://docs.servicenow.com/csh?topicname=c_DocumentScoring.html&version=latest In the customer's case they did not have inverse document frequency (IDF) enabled. Release Jakarta

Cause After engaging with development it was identified that customer was encountering the following problem:

PRB1237035 Knowledge results always appear above other categories because its ir_query_score is consistently higher

This is fixed in London Release.

Resolution The workaround was to set the following attribute on the kb_knowledge table's collection entry in the dictionary:

text_index_attachment_displayed = true

This attribute is currently undocumented.

Development advised "That attribute (text_index_attachment_displayed) specifies if attachments will be displayed in text search results."

The folloiwng documentation enhancement request has been raised:

DOC61116 collection attribute text_index_attachment_displayed

Note: It was not necessary to re-regenerate the text index for the kb_knowledge table after making the above change.

After adding the attribute the customer was able to see the most relevant articles at the top of the search results.

Debug of the text search revealed that the scoring was no longer being capped at the value 255:

other (not gif, sql)12:03:09.142: >>>> other (not gif, sql)12:03:09.145: 1: Score for document e14fdf2bdbf91b00afe666a25b9619bd is: 510.0 other (not gif, sql)12:03:09.145: 2: Score for document ec7943a3dbb91b00afe666a25b9619e5 is: 505.0 other (not gif, sql)12:03:09.145: 3: Score for document 1f8e93e7dbf91b00afe666a25b961919 is: 500.0 other (not gif, sql)12:03:09.145: 4: Score for document 1631afebdbf91b00afe666a25b9619bd is: 255.0 other (not gif, sql)12:03:09.145: 5: Score for document 876fdbe7dbf91b00afe666a25b96193a is: 255.0 other (not gif, sql)12:03:09.145: 6: Score for document 4c5e17a7dbf91b00afe666a25b96199d is: 252.0 other (not gif, sql)12:03:09.145: 7: Score for document af7ddb27dbf91b00afe666a25b961969 is: 250.0 other (not gif, sql)12:03:09.145: >>>>