How to Identify Slow Lookup Variables in the Service CatalogIssue This article provides a method of identifying "Lookup Select Box" variables that may cause slowness due to excessive lookups. This slowness may be incurred/observed in various places: Initial loading of a catalog itemDisplaying a service catalog task (sc_request, sc_req_item, sc_task, etc.) where the form includes the Variable Editor formatterText Indexing of a service catalog task You may see a stack trace similar to the following, which will indicate that you are hitting this type of issue: main,Default-thread-109,5,attrs=(session_id=8D3C0B18DB1F3F00D3C39EB5DB9619F3) java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(SocketInputStream.java:116) java.net.SocketInputStream.read(SocketInputStream.java:171) java.net.SocketInputStream.read(SocketInputStream.java:141) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:82) org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:92) org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket(RawPacket.java:77) org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket(SyncPacketFetcher.java:67) org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:966) org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:1076) org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:1055) org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:295) org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:393) sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) com.glide.db.StatementWrapper.invoke(StatementWrapper.java:40) com.sun.proxy.$Proxy7.execute(Unknown Source) com.glide.db.DBI.executeStatement0(DBI.java:1053) com.glide.db.DBI.executeStatement(DBI.java:1009) com.glide.db.DBI.executeStatement(DBI.java:970) com.glide.db.DBI.executeStatement(DBI.java:958) com.glide.db.DBQuery.executeAsResultSet0(DBQuery.java:376) com.glide.db.DBQuery.executeAsResultSet0(DBQuery.java:351) com.glide.db.DBQuery.executeAndReturnTable(DBQuery.java:308) com.glide.db.DBAction.executeNormal(DBAction.java:236) com.glide.db.DBAction.executeAndReturnException(DBAction.java:190) com.glide.db.RDBMSQueryContext.executeQuery(RDBMSQueryContext.java:46) com.glide.db.DBQuery.execute(DBQuery.java:2636) com.glide.db.meta.Table.queryBasic(Table.java:315) com.glide.db.meta.Table.query(Table.java:201) com.glide.script.GlideRecordITable.query(GlideRecordITable.java:101) com.glide.script.GlideRecord.query0(GlideRecord.java:3256) com.glide.script.GlideRecord.query(GlideRecord.java:2981) com.glideapp.questionset.LookupSelectQuestion.loadChoiceList(LookupSelectQuestion.java:102) com.glideapp.questionset.LookupSelectQuestion.getChoiceList(LookupSelectQuestion.java:46) com.glideapp.questionset.AbstractChoiceListQuestion.getDisplayValue(AbstractChoiceListQuestion.java:35) com.glideapp.questionset.AbstractChoiceListQuestion.getDisplayValue(AbstractChoiceListQuestion.java:26) com.glideapp.questionset.LookupSelectQuestion.getDisplayValue(LookupSelectQuestion.java:58) com.glide.service_portal.variables.VariableModel.setValue(VariableModel.java:710) com.glide.service_portal.variables.VariableModel.augmentByType(VariableModel.java:539) com.glide.service_portal.variables.VariableModel.buildVariable(VariableModel.java:311) com.glide.service_portal.variables.VariableModel.get(VariableModel.java:140) com.glide.service_portal.catalog.SPCatalog.getItem(SPCatalog.java:188) com.glide.service_portal.catalog.SPCatalog.getItem(SPCatalog.java:48) com.glide.service_portal.catalog.SPCatalog.getItem(SPCatalog.java:44) com.glide.service_portal.widget.SPScriptable.getCatalogItem(SPScriptable.java:957) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138) org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:292) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2585) org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292._c_anonymous_2(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script:54) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292._c_anonymous_1(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script:2) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292._c_script_0(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script:1) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script) org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563) org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3428) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script) org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.exec(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script) com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279) com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118) com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82) com.glide.service_portal.SPScriptEvaluator.eval(SPScriptEvaluator.java:37) com.glide.service_portal.widget.SPScriptRunner.eval(SPScriptRunner.java:221) com.glide.service_portal.widget.SPScriptRunner.runScript(SPScriptRunner.java:96) com.glide.service_portal.widget.SPWidget.get(SPWidget.java:123) com.glide.service_portal.widget.SPWidget.get(SPWidget.java:106) com.glide.service_portal.page.SPPageLayoutCache.getColumnRectangles(SPPageLayoutCache.java:49) com.glide.service_portal.page.SPPageLayout.getRowColumns(SPPageLayout.java:115) com.glide.service_portal.page.SPPageLayout.getContainerRows(SPPageLayout.java:82) com.glide.service_portal.page.SPPageLayout.getContainers(SPPageLayout.java:56) com.glide.service_portal.page.SPPageLayoutCache.getContainers(SPPageLayoutCache.java:34) com.glide.service_portal.page.SPPageCache.getContainers(SPPageCache.java:26) com.glide.service_portal.page.SPPage.buildPageResponse(SPPage.java:94) com.glide.service_portal.page.SPPage.getPage(SPPage.java:81) com.glide.service_portal.page.SPPage.get(SPPage.java:57) com.glide.service_portal.SPRestService.page(SPRestService.java:80) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:43) com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:286) com.glide.processors.AProcessor.runProcessor(AProcessor.java:532) com.glide.processors.AProcessor.processTransaction(AProcessor.java:230) com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:178) com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:167) com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31) com.glide.sys.Transaction.run(Transaction.java:2091) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) The "LookupSelectQuestion" keyword indicates that this is a "Lookup Select Box"-type variable. There are several solutions to speeding up these variables: Convert the variable from type "Lookup Select Box" type to a "Reference" type variable. This will change the user experience from using a listbox to using a "Lookup Using List".Add a more selective filter to the reference qualifier for the variable.Use an appropriate* source reference table. * An inappropriate source reference table would be task or incident. An appropriate source reference table would be the Choices [sys_choice] table. For example, if you had an incident record producer, and that had a variable named "category", it would be inappropriate to query the entire incident table to get all of the category values. Instead, the reference table should be set to the sys_choice table and a reference qualifier should specify the table and field (eg., name=incident^element=category). Procedure Navigate to the System Maintenance > Scripts - Background module, copy the following snippet of javascript into the field labeled, "Run script (JavaScript executed on server)", and click on the "Run Script" button. var MAX_LOOKUP_RECORDS = 1000;var gsuri = gs.getProperty('glide.servlet.uri');var stdout = ['\n'];// Query the Variables table for var gr = new GlideRecord('item_option_new');gr.addQuery('type', '=', 18);gr.query();while (gr.next()) { try { // Check to make sure the lookup_table is defined if (gr.lookup_table) { // Determine how many records the Lookup Select Box will return var count = new GlideAggregate(gr.lookup_table); if (count.isValid()) { if (gr.reference_qual) { var encoded_query = null; if (gr.reference_qual.startsWith('javascript')) { if (gr.reference_qual.includes('current.variables')) { continue; } var evaluator = new GlideScopedEvaluator(); encoded_query = evaluator.evaluateScript(gr, 'reference_qual', null); } else { encoded_query = gr.reference_qual; } if (!encoded_query) { continue; } count.addEncodedQuery(encoded_query); } count.addAggregate('COUNT'); count.query(); var lookupRecords = 0; if (count.next()) { lookupRecords = count.getAggregate('COUNT'); } // Report a warning if the number of matching records exceeds the MAX LOOKUP RECORDS if (lookupRecords > MAX_LOOKUP_RECORDS) { stdout.push('Variable: ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n'); stdout.push('\tLookup Table: ' + gr.lookup_table); stdout.push('\tReference Qualifier: ' + count.getEncodedQuery() + '\n'); stdout.push('\tThere are ' + lookupRecords + ' matching records\n' + '\tLIST URL = ' + gsuri + gr.lookup_table + '_list.do?sysparm_query=' + count.getEncodedQuery() + '\n'); } } else { // not a valid table stdout.push('Variable - ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n'); stdout.push('\t*** Invalid Variable: The lookup_table value for this variable ( ' + gr.lookup_table + ' ) does not match not a valid table.\n '); } } else { stdout.push('Variable - ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n'); stdout.push('\t*** Invalid Variable: The lookup_table value for this variable IS NULL.\n '); } } catch (err) { // An error occurred; report something helpful var st = err.getStackTrace(); stdout.push('Variable - ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n' + 'Stack Trace:\n'); for (var i = 0; i < st.length; i++) { stdout.push(st[i].toString()); } stdout.push('\n\n'); }}gs.print(stdout.join('\n')); You can adjust the value of the MAX_LOOKUP_RECORDS variable in the script to suit your purposes; however, as a best practice, a Lookup Select Box should never return more than 100 records. Example Output The script will provide output such as the following: *** Script: Variable - department https://myinstance.service-now.com/item_option_new.do?sys_id=e70ec99e53032200eb7c0a1806dc3449 There are 100002 matching records LIST URL = https://myinstance.service-now.com/sys_user_list.do?sysparm_query= Variable - table https://myinstance.service-now.com/item_option_new.do?sys_id=e849da79471102003db6d7527c9a7128 There are 3189 matching records LIST URL = https://myinstance.service-now.com/sys_db_object_list.do?sysparm_query= Variable - broccoli https://myinstance.service-now.com/item_option_new.do?sys_id=e849da79471102003db6d7527c9a7128 Invalid Variable: The lookup_table value for this variable IS NULL. Variable - meowmeow https://myinstance.service-now.com/item_option_new.do?sys_id=e849da79471102003db6d7527c9a7128 Invalid Variable: The lookup_table value for this variable (meownow ) does not match not a valid table. The output first shows the name of the variable and the link to the variable definition. Next, it shows the number of rows that would be returned by the lookup for this variable. Finally, it provides a link to the list of the data that would be used by the variable. The LIST URL will likely render quickly, because we are usually only displaying the first 1-20 rows; however, keep in mind that the variable lookup would not paginate; it would retrieve all records. The output also shows errors if the lookup table is empty, is invalid, or for some other reason we are unable to get the rowcount of the variable.ReleaseAllRelated LinksCreate a service catalog variable Service catalog variable attribute Types of service catalog variable