How to troubleshoot Slow Transactions or Page Load Issues in Next Experience Workspace and Portal ExperiencesOverview This article is aimed at ServiceNow customers and partners who are trying to solve performance issues (e.g., slow response times or page load failures) in Workspace Experiences. Before bringing your issue to ServiceNow support, it can be very helpful to see if you can troubleshoot it yourself. This has two benefits. First, you might solve the issue yourself, saving time and making you look like a hero (well done!). Secondly, if you aren't able to completely solve the issue, you will have solid information that you can provide to ServiceNow support when you open your case. Table of Contents OverviewA Good Problem StatementCommon Types of Failing TransactionsClient Interactions How Client Interaction Total UI Time WorksSeeing the Client Interactions for just one Application Transaction LogsNED Tools (Next Experience Developer Tools) A Good Problem Statement The most helpful thing that you can do when opening a Case with Support is to have a clear problem statement. A good problem statement may include some or all of the following items: Historical examples of the issues so ServiceNow can immediately find the relevant logs 👈 THIS IS THE MOST HELPFUL THING YOU CAN GIVE US!! THIS IS OFTEN ENOUGH TO SOLVE AN ISSUE ALL BY ITSELF Ideally a link to the relevant transaction (syslog_transaction) or interaction (sys_client_interaction) record in your instanceIf not, then include user names, actions they were taking, URLs, screenshots, and exact times (please, please, please include the timezone 😬 - a great tool for Timezone conversions and documentation is https://www.timeanddate.com/worldclock/converter.html) A clear description of the overall issue. What is the problem? Include expected vs. unexpected behavior. How slow is slow, 3 seconds or 3 minutes?Where is this observed? Environments, versions, workspaces, applications, mobile, desktop, virtual machines, regionsWho is impacted? Is it a certain group of users? Please provide exact user names of users who have complained so ServiceNow can look at their logs.When does it happen? All day or just certain times? When did it seem to start? Has it been getting worse? A clear description of what is NOT the issue What is NOT the problem?Where is this NOT observed?Who is NOT impacted?When does it NOT happen? Distinctions between different flavors of the issue. You might actually have multiple issues, so being clear about the different symptoms will help clarify the investigation.Steps to reproduce the issue Complete: Include all steps from the time of login, don't assume someone will know that you've skipped a stepDiscrete: Include only one sequential step at a time.Concrete: Use language that describes real actions (e.g., "Click the Submit button in the header of the Case form" rather than "Submit a new Case"))Also include pre-requisites (e.g., turning on a plugin, what environment, what user to impersonate etc.) Good ExampleLess Good (AKA Bad) Example We have been seeing periodic slow page loads (many over 20 seconds) in the Software Asset Workspace (go to https://acme.service-now.com/now/softwareasset/home) since around mid January. There are many examples of this issue - maybe, 3 or 4 per day. We can't look into all of them, but one example we looked at was user Abel Tutor (abel.tutor) who waited over 30 seconds while trying to view unassigned Assets on Feb 2, 2024 at around 11:00 US Central (UTC -6). We followed the steps to find the related Transaction logs, but no transactions with response_time over a couple seconds were found. Steps to reproduce Impersonate abel.tutor Open Software Asset Workspace Click the lists icon from the left nav Try to open the "Unassigned Assets" list Query we used to find example transactions: https://acme.service-now.com/syslog_transaction_list.do?sysparm_query=sys_created_onBETWEENjavascript:gs.dateGenerate('2024-02-02','10:50:00')@javascript:gs.dateGenerate('2024-02-02','11:20:00')^sys_created_by=abel_tutor The users are raising ongoing performance issues on workspaces in AM area. Common Types of Failing Transactions There are a two basic types of failing transactions. While these issues typically look similar, they have very different causes. For example, suppose your symptom is a blank white screen after you click a button. This could be caused by a page that is breaking (some logic is failing and thus not loading as expected) or a page that is running slowly (the page is loading, but it just takes a long time, maybe even so long that it never loads/times out). Each have very different causes/solutions, so it is important to understand which one you are dealing with. Pages that break - i.e., a type of functional issue where pages don't load at all or they render incorrectly Causes include the following: Race conditions during client-side rendering (the Next Experience Component Event processing is inherently non-deterministic/asynchronous - see https://developer.servicenow.com/dev.do#!/reference/next-experience/washingtondc/ui-framework/main-concepts/code-execution)Conflicting or broken themesA business logic failure on the server that resulted in a failed transaction. These usually shows up as a "Internal Server Error 500...".A transaction that gets cancelled due to another transaction from the same user that came later. This also shows as "Internal Server Error 500". Cancelled transactions are often impacted/caused by a key feature of ServiceNow called session synch - see KB0755706 - What is Session wait / Session Syncronisation? Pages that load slowly Causes include the following: Inefficient database queries (See Best Practice for Building Efficient Queries)Excessive server-side processing (See Performance Best Practices for Server-side Coding in ServiceNow)Large pages/resources getting downloadedExcessive client-side processing (e.g., Repeater component with 1,000 results, see Performance Hacks: Next Experience "Repeater" and "Pagination control" Components)Memory shortage on the browserMemory or processing bottlenecks on the Operating SystemNetwork lag/failures (See Troubleshooting Network Performance and How to check GraphQL network calls in Agent Workspace)A transaction that gets cancelled because it ran longer than the quota NOTE: A really slow transaction that times out and/or gets cancelled can look similar to a page that doesn't load at all, but the causes will be similar to other slow loading pages, rather than other pages that don't load at all. Many time, the causes of Next Experience failures are the same as ServiceNow's Core UI. You should familiarize yourself with the following article which includes a video tutorial KB0997495 - How to troubleshoot a slow transaction. Client Interactions The Client Interactions table (sys_client_interaction) stores information about, well... interactions. Interactions track how a user navigates within a ServiceNow website. In modern web development, when you click a button there may be hundreds of individual, overlapping transactions. So how do you measure how long it took for the page to load after you clicked a button? How do you know which transactions were related to your button push and which were not? An Interaction aggregates all those transactions into one place. But it does more than that. It also keeps track of the total time it takes from start to finish. This is ServiceNow's version of what the industry calls Real User Monitoring (RUM). As of the Washington DC release, Client Interactions are still a work in progress, but they have become increasingly robust and accurate since the Tokyo release. By Xanadu release, many important defects and enhancements will be complete, with more planned over the next two releases. There are many useful fields in the Client Interactions table, but the most important is probably the total_ui_time field. Total_ui_time tracks all the time between a page being requested and that page being considered "idle" (see detailed description of what idle means in the section below). It gives you a picture of what a user is really experiencing. Two other useful fields, related to total_ui_time, are ui_time_before_load and ui_time_after_load. The "load" event is essentially when the first skeleton of a page has started rendering in the browser. All the time between the start of an interaction and "load" is ui_time_before_load. All the time between "load" and "idle", is ui_time_after_load. The following fields might be useful to add to your Client Interaction ID list layout. How "Total UI Time" Works (Beware: technical mumbo jumbo!) Seismic, the Next Experience component framework, has a scheduler for event types; render, effect (actions), dispatch, state change, prop change, etc. Seismic also has a network queue. We pay attention to the scheduled events by component and outbound network requests, if we see that both are idle, then Seismic sends out a promise that says Seismic is idle. UXF (the Server-side framework that enables the declarative metadata and runtime of Next Experience), because it understands composition of the page, waits to pay attention to that. When it sees Seismic is idle, it verifies that the page level Macroponent is actually settled. As soon as both UXF and Seismic agree that nothing else is ongoing, we trigger a promise that captures the current timestamp and this is how we determine Total UI Time (sys_client_interaction.total_ui_time). One thing to note is that, as of Xanadu, Client Interaction does not work on every little click. For example, it does not capture navigations within a single tab, like clicking the magnifying glass icon to search for records for a reference field. It only works on hard refreshes of the browser (“page” load), any time you directly type a URL into the browser to navigate, or if there is a "NAVIGATION" event - which, as of Xanadu, only includes things like loading a list, or loading or submitting a record, or page. Seeing Client Interactions per "Application" Another useful field in Client Interactions is the "Application field. The field "Application" can be used to see, for example, only those Interactions from the "HR Agent Workspace". You may need to personalize your list layout using the List Mechanic (cog wheel icon) to show the Application field on the list. If you have multiple reports of issues with a certain Workspace or you just want to find out the health of one of your Workspaces, you can use the Client Interaction table to investigate. You could do something very simple, like just look directly at a list of Client Interaction records, or you might want to schedule fancy weekly reports that show aggregate load times to keep you informed. Understanding "Screen Routes", "Screen Fields" and "Screen Params" In Next Experience, a Screen is basically analogous to a page. You can have a screen load as a modal or you can have it load as a sub-page or you can load it in a totally new browser tab. Each of these ways of accessing a Screen can be a different Route. The Route is the argument immediately following the Workflow path. To pass arguments to a Screen, some details can be included in the URL. Fields are required arguments and Params are optional arguments. Transaction Logs Transactions Logs have been a staple diagnostic tool in ServiceNow for many years. Transactions Logs are still relevant for Next Experience based transactions. One tricky thing is that the URL names are not self explanatory like they were with previous UI versions. For example, in the Core UI transactions have URLs like incident.do and incident_list.do, but most transactions in Next Experience have URLs like /api/now/par/pardata, /api/now/graphql or /api/now/batch. In order to understand more about what each of these do you need to use additional tools like the following: Inspect the Node Logs.Use a client-side tool like NED Tools or Chrome Developer ToolsEnable ServiceNow Session Debugging that will output more details to the UI. One helpful feature of Client Interactions is that at the bottom of each Client Interaction form, there is a Related List that includes most, if not all, of the related Transaction Log records for that Interaction. For a description of all fields and how they work, see "Troubleshooting Guide: Using the Transaction Logs" https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0584420 Also, as mentioned earlier, for a video on how to use the logs, "How to troubleshoot a slow transaction" https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0997495 You will want to personalize your list view of the Slow Transactions table. Here is an example of some helpful fields you may want to add: NED Tools (Next Experience Developer Tools) NED Tools, previously known as Seismic Developer Tools, is your Swiss army knife for troubleshooting any type of Next Experience issue. Most notably, it has an "Inspector" feature and a "Profiler" feature. The "Inspector" allows you to drill down to the specific Components in the UI and see their real-time related Properties, Actions and Transactions. The "Profiler" allows you to record, save and compare a period of Next Experience activity. Troubleshooting Scenarios with NED Tools - KB1543474 (covers Analyzing Performance with Flamecharts, Comparing with the Profiles Pane and much more)https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1543474 User Manual in ServiceNow Dev sitehttps://developer.servicenow.com/dev.do#!/reference/next-experience/latest/developer-tools/using-next-experience-developer-tools Next Experience Developer Tools Release Notes on ServiceNow Doc sitehttps://docs.servicenow.com/csh?topicname=nedt-rn.html&version=latest