dictionary attribute 'Text index tokenizer language' is not working for some Japanese Katakana characters - Known Error

dictionary attribute 'Text index tokenizer language' is not working for some Japanese Katakana characters Description dictionary attribute 'Text index tokenizer language' is not working for some Japanese Katakana characters

Steps to Reproduce

1. log into instance

2. go to kb_knowledge.list, and then new one

3. input Knowledge base='knowledge' and short description = 'インターネットメニュー・アプライアンスメニュー', then submit

4. do text search for 'アプライアンス', confirmed record can be picked up

5. from dictionary of kb_knowledge, add dictionary attribute 'Text index tokenizer language' , and then re-generate text index following below doc

https://docs.servicenow.com/bundle/orlando-platform-administration/page/administer/search-administration/task/configure-tables-japanese-tokenizer.html

6. do text search for 'アプライアンス' again, confirmed record is not picked up now

7. can confirm text search now only works for 'アプライアンスメニュ', ideally 'アプライアンスメニュー' should be separated as 'アプライアンス' and 'メニュー'

Workaround It's working as expected as dictionary used by tokenizer won't be perfect always.

User is able to maintain the text indexing word by below steps

1. add sys_properties glide.ts.japanese.enable.user_dictionary=true 2. run ts_japanese_token_dictionary.list, and new a entry ’アプライアンス’ 3. re-index the target table