CHAT slowness reported by users - Support and Troubleshooting

CHAT slowness reported by users Issue  Users are reporting slowness when accessing the chat tools.

Log files show excessive wait times, high volumes of conversation records and even high semaphore waits.

Examples:

2019-07-24 08:34:03 (026) Default-thread-7 9957C9DADB7E3F4439FAD211CE9619B4 txid=9b9fa916db7e EXCESSIVE *** End #4550813 /api/now/live_feed/conversations, user: USER1, total time: 0:00:05.445, processing time: 0:00:05.322, total wait: 0:00:00.123, session wait: 0:00:00.123, SQL time: 0:00:01.754 (count: 1,098), business rule: 0:00:00.243 (count: 459), ACL time: 0:00:00.955, source: xxx.xxx.xxx.xxx , type:rest, method:POST, api_name:now/live_feed, resource:now/live_feed/conversations, version:Default, user_id:user1, response_status:200

2019-07-25 06:31:35 (825) Default-thread-6 CA549AA2DB36FF84E92B3B98F4961900 txid=5f2d9662db7a EXCESSIVE *** End #9975855 /api/now/connect/conversations/.../ actions, user: JOE.USER@HELP.com, total time: 0:00:05.482, processing time: 0:00:01.248, total wait: 0:00:04.234, session wait: 0:00:01.342, semaphore wait: 0:00:02.892, SQL time: 0:00:00.994 (count: 191), business rule: 0:00:00.590 (count: 62), ACL time: 0:00:00.003, source: xxx.xxx.xxx.xxx , type:rest, method:GET, api_name:now/connect, resource:now/connect/ conversations/{conversation_id}/actions, version:Default, user1, response_status:200

2019-09-10 07:42:11 (503) Default-thread-9 BDB548F51B3F7B8016A64332DD4BCBA7 txid=a700bc311bb7 EXCESSIVE *** End #6748696 /api/now/connect/conversations/1947e8f51bf3b7807c99fd12dd4bcb93, ... semaphore wait: 0:00:03.444, SQL time: 0:00:02.636 (count: 27), business rule: 0:00:00.009 (count: 10) , type:rest, method:GET, api_name:now/connect, resource:now/connect/conversations/{conversation_id}, version:Default, user_id:user, response_status:200 Release All releases

Cause The chat tool is designed to be customizable and unlimited. This will create a performance impact over time, that will continue to get worse without getting cleaned up. PRB1274858 - When users have a large number of open chats, they experience slowness after logging in

Resolution Initial Clean-up will be needed and may take time to run, be sure to test in sub prod or against a clone first.

A. For a SINGLE USER:

Conversations impacting one specific user as well as high memory: 1. Change the user name from "EXACT USER NAME" to the one that needs relief 2. Run the script and see if the output shows 'Updated records: ' more than 50. If yes then follow below steps 3 and 4 3. Remove the commented 3 lines after count++ to update the records 4. Run the script from scripts.do

var gr = new GlideRecord('live_group_member'); gr.addQuery('member.name=EXACT USER NAME^state!=inactive^visible=true^sys_created_on gr.query(); var count = 0; while (gr.next()) { count++;

//Un-remark the following out when you are ready to fix one user //gr.state='inactive'; //gr.visible=false; //gr.update(); } gs.print('Updated records: ' + count);

B. CLEANUP of all users:

1. This only updates conversations that have not been viewed since last week and if the user has more than 50 (if they have less than that they may not be facing the issue), but that number can be changed on the addHaving line if. 2. This only sets visible field to false, but if all of the conversations are support related you might also want to change the state field to inactive. 3. Be sure to create a scheduled job so that it would run every so often, but the first time please make sure you run it during off hours and test in a sub-prod since the update can take a while. Can take days for first cleanup. 4. Since every customer has a different use case for their chat data, please do not run the following without validating the data

var member = new GlideAggregate('live_group_member'); member.addQuery('visible=true^member_type=user^last_viewed member.addAggregate('COUNT', 'member'); member.groupBy('member'); member.addHaving('COUNT', '>', 50); member.query(); gs.print('Members with 50+ visible Chats:' + member.getRowCount()); var totalCount = 0;

while (member.next()) {

var userCount = 0; var gr = new GlideRecord('live_group_member'); gr.addQuery('visible=true^member_type=user^last_viewed gr.addQuery('member', member.member); gr.query();

while (gr.next()) { userCount++; totalCount++;

//uncomment the lines below to update the records after verifying the results look correct //gr.state='inactive'; //gr.visible=false; //gr.update() } gs.print('From Agg: ' + member.member.getDisplayValue() + '--Updated Record Count: ' + userCount); } gs.print('Total records that were updated: ' + totalCount);

C. A scheduled job will need to be created for this as well here are the options:

There are 3 potential workarounds. Please choose the one that fits your need...

1. In order to prevent the loading of the excessive chat conversations, you can edit the live_group_member records for the affected user. To accomplish this, find the affected user's live_group_member record and manually set the State field to Inactive, and the Visible field to false.

NOTE: If the user needs some of their Connect conversations to stay active and visible, then you will need to find a way to limit them before setting the visibility and state. 2. The attached 'Workaround_Script_PRB1274858' is the scripted workaround to set live_group_member state to inactive and visible to false.

Here are the variables to be considered within the script...

targetChatCount: Is the target for the max. number of active conversation memberships a user should have. In script value is 50, update it according to your requirement if needed.

deltaTime: How long should have passed since a user last viewed (or since the conversation was opened if it has never been viewed) before we consider closing the corresponding membership. Value is 14 days, update it according to your requirement if needed.

hardClose: By default the script will leave a user with > targetChatCount if many of their conversation memberships are 'recent' (within deltaTime) – Setting this forcefully gets all users down to targetChatCount memberships.

dryrun: Set to active (1) in the script. Make sure to disable dryrun (0) to actually execute the script after a test run. NOTE: Please test the script in subprod instances and make sure to run the dryrun first to note the changes this script will make.

3. If performance issue is recurring, this is the 3rd option you can use. This workaround is to add scheduled job and cleanup chat conversations related to 'Task records which are inactive for more than 14days'. Please modify the script in the schedule job or script if needed to do cleanup on a different criteria.

a. Import the update set attached "sys_remote_update_set_e11e53c2db1cf38023a3581bdc9619b3 (1)". Committing update set will insert: Scheduled Job (sysauto_script_407d53c2db1cf38023a3581bdc9619a1) and Database View (sys_db_view_f7ae4f06dbd8f38023a3581bdc961930)

b. Run the attached script "Workaround_Script_2_PRB1274858" from scripts background after committing update set.

c. Make sure to activate the schedule job (sysauto_script_407d53c2db1cf38023a3581bdc9619a1) after running the script from background script.

NOTE: Please test any workaround in sub prod instances and make sure to run the dryrun first to note the changes this script will make.

Related Links KB0634397 - FAQ for Connect Chat and Connect Support

KB0677335 - Connect Chat Performance Issues: Overview and Best Practices

KB0635589 - Connect chat/support blank page or loading issues