MID Server and Code Signing - Unable to find the signature / Cannot just-in-time load the signature record / Failed to verify signature - Support and Troubleshooting

MID Server and Code Signing - Unable to find the signature / Cannot just-in-time load the signature record / Failed to verify signature .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } This KB article aims to explain the Errors seen in the syslog table and appnode logs for exceptions related to Code Signing of records, when fetched by various features' probes running in the MID Server platform.

Unable to find the signature with the sys id, X, for the table, Y.: com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record:Couldn't find pluginId for documentId: X with signatureRecordSysId: Z: com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) ...

There are several variations of the table name, and stack trace, depending on the feature and records involved, and retrieval method.

Table of Contents How the MID Server works How the instance side works sn_kmf_record_signature The change in Washington DC System Properties com.snc.kmf.signature.validation.flag com.snc.kmf.signature.use.jit_loading com.snc.kmf.cert.validation.enabled com.snc.kmf.signature.validation.optin MID Server Script Includes (ecc_agent_script_include) Signing your own scripts MID Server Attachments (ecc_agent_attachment) MID Server Synchronised Files Script Files (ecc_agent_script_file) MIB Files (ecc_agent_mib) ACC Plugins (sn_agent_asset) JAR Files (ecc_agent_jar) Discovery Pattern Uploaded File (sa_uploaded_file) ACC Config file (sn_agent_configuration_file) When the Table is not mentioned, and other Non-MID Server-related errors Restricted Caller Access Flow Engine Solution System Exports Sets (ecc_agent_attachment) Agent Client Collector (sn_agent_asset, sn_agent_configuration_file) Discovery IntegrationHub Synthetic Monitoring How the MID Server works Since the Utah version, when the MID Server needs any MID Server Script include, and other tables, it always retrieves the script from the instance using SignedGlideRecord. With SignedGlideRecord, the MIDServer retrieves both the script and its corresponding signature from the instance.

If code signing is active, the MIDServer uses the signature information. Even if Code Signing isn't enabled in the instance, in certain situations the MID Server will still need the scripts' signatures. For example when securing credentials on the MID Server side.

However when code-signing is disabled, the MID Server should not need to do this, and so this Problem exists, fixed in Australia: PRB1892912 / KB2763600 MID Server tries to download signatures for synced file attachments when code signing is disabled

This problem occurs because getAttachmentMetadata() in InstanceAttachment.java does not honor the 'signatureRequired' method parameter and always passes true for the 'need_signature' parameter in the MIDServerFileProvider request.

A MID Server issues record in the instance will be created, even when code signing is not enabled.

How the instance side works All out-of-box signatures are signed in the build process, so when a feature's table requires code signing, the relevant ServiceNow development team need to mark that table for code signing in our build system.

That provides the ability for those records/resources to now be signed using that signature, however they do not get loaded until the resource is used. This is provided by a feature we call JIT (Just In Time Loading).

Code Signing does not need to be active in order for signatures to be loaded , but it is necessary for signatures to be verified before usage on the platform. This means loading of signatures is still expected in instances that do not have Code Signing enabled. If that loading fails, it is an indication of a problem, even if code signing is not enabled, but if it is not then the error can be safely ignored.

sn_kmf_record_signature The KMF Signature Records table [sn_kmf_record_signature] stores the signature for each record that has one. The Table Name plus Document ID fields specify which record the signature is for.

The change in Washington DC Since WashingtonDC, the Code Signing code in the instance was updated to add a new 'Map' and also added the error message we see.

When MID Server sends a request to retrieve a record from the instance: - Before W: If the signature is not available, the code returns null, but without any error message - from W: If the signature is not available, the instance side Code Signing APIs still return null, but also cause a Severe Error from an exception.

The return value is the same before and after the W release, so the MID Server and any probes/features using those records/scripts/attachments work as before, so this shouldn't break any MID Server-related features , and can be ignored.

These occurs in instances that don't have Code Signing properties enabled. The error is for Loading the signature, not for the more important Verification if code signing were enabled, but is still a valid error.

System Properties Key Management Framework, and Code Signing framework plugins are installed in all WashingtonDC instances. Code Signing is usually not enabled, and is not by default. If code signing is enabled on a Production instance then the customer will have had to have requested this from customer support in the past.

Docs link: Properties installed with Code Signing

com.snc.kmf.signature.validation.flag Returns whether signature validation is on/off for the instance.

com.snc.kmf.signature.use.jit_loading When this property is true, verification happens by loading signature record on demand. This property can only be modified by maint (which means customers can't), and defaults to True .

The code giving the error will run only if this is set True, even if com.snc.kmf.signature.validation.flag is False.

com.snc.kmf.cert.validation.enabled This property, if true, means that when MID server validates signatures, they will also validate the certificate chain that was used to sign the signature. This property was introduced in V, and is turned to true automatically when code signing property (com.snc.kmf.signature.flag) is turned on.

com.snc.kmf.signature.validation.optin When true, enables Code Signing on the instance.

MID Server Script Includes (ecc_agent_script_include) The following features, and possibly more, use MID Server Script Includes:

The MID Server platform itself Discovery and Service Mapping Cloud Provisioning and Governance Agent Client Collector Health Log Analytics custom JavascriptProbe integrations These are Javascript scripts for execution in the MID Server platform (so are allowed to include Packages calls).

When a probe running in the MID Server needs any MID Server Script include, which is not already available in its cache, it will fetch it using REST (getScriptFromDB in MIDScriptIncludes.java).

This example error accounts for 2/3 of all errors seen, and is the CL_OciInvoker script of the Cloud Access API used by the Cloud Operations Workspace. The sys_id/documentId will be different for each record.:

2024-06-07 02:54:47 (983) API_INT-thread-3 E876007E1BEE4A14AEE78622604BCB7B txid=daef4b831b6a LocalhostTransactionLogger *** Start #3808620 /api/mid/v2/glideRecord/ecc_agent_script_include, type: rest, user: X ... 2024-06-07 02:54:47 (988) API_INT-thread-3 E876007E1BEE4A14AEE78622604BCB7B txid=daef4b831b6a SignatureUtil DEBUG: Could not find the KMF Signature record using fields: document_id : 00f196cdb7682010d53b32c6ee11a938 table_name : ecc_agent_script_include purpose : mid_script ... 2024-06-07 02:54:47 (997) API_INT-thread-3 E876007E1BEE4A14AEE78622604BCB7B txid=daef4b831b6a SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, 00f196cdb7682010d53b32c6ee11a938, for the table, ecc_agent_script_include . com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: 00f196cdb7682010d53b32c6ee11a938 with signatureRecordSysId: 4e147ae9547f4a3c985237882010ff10 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordGRUsingFields(CodeSigningOutputUtility.java:120) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningSignatureRecordForRestEndPoint(CodeSigningOutputUtility.java:85) at com.snc.core_automation.security.validation.application.SignatureUtil.loadSignatureRecord(SignatureUtil.java:115) at com.snc.core_automation.security.validation.application.SignatureUtil.getSignatureV2AsMap(SignatureUtil.java:80) at com.snc.core_automation.ws.inbound.GlideRecordRESTServiceV2.getSignature(GlideRecordRESTServiceV2.java:149) at com.snc.core_automation.ws.inbound.GlideRecordRESTServiceV2.getRecords(GlideRecordRESTServiceV2.java:63) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:44) at com.glide.rest.processors. RESTAPIProcessor.process(RESTAPIProcessor.java:345) ... Signing your own scripts See KB1516626 How to sign or re-sign a MID Server Script Include without activating code-signing

MID Server Attachments (ecc_agent_attachment) The following features, and possibly more, use MID Server Script Includes as part of their probes:

System Export Sets This error will occur each time a StreamPipeline probe runs for an Export Set, and fetches the attachment via the MIDServerFileProvider scripted SOAP service.

Example log. The sys_id will be different every time, as the temporary ecc_agent_attachment records are created at runtime.

2024-06-07 03:00:05 (992) SOAPProcessorThreadb311db0b93ea4a1c363fb34a6aba10a5 36C41E5693E242D8363FB34A6ABA1038 txid=3711530b93ea SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, 17119f83c32a4ad874ba99da0501316f, for the table, ecc_agent_attachment . com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: 17119f83c32a4ad874ba99da0501316f with signatureRecordSysId: 3004386fdfb94d9792869cc38e8a4d01 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordGRUsingFields(CodeSigningOutputUtility.java:120) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningSignatureRecordForRestEndPoint(CodeSigningOutputUtility.java:85) at com.snc.core_automation.security.validation.application.SignatureUtil.loadSignatureRecord(SignatureUtil.java:115) at com.snc.core_automation.security.validation.application.SignatureUtil.getSignatureV2AsMap(SignatureUtil.java:80) at com.snc.core_automation.security.validation.application.SignatureUtil.getVersionedSignatureAsJSON(SignatureUtil.java:136) at com.snc.core_automation.security.validation.application.SignatureUtilJS.jsFunction_getVersionedSignature(SignatureUtilJS.java:53) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:151) at org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:717) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:654) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2973) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1720) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:1030) at org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:161) at com.glide.caller.gen. sys_web_service_6c94a96683201000dada83ec37d929c8_script .call(Unknown Source) ... at com.glide.processors.soap.SOAPCustomProcessor.execute(SOAPCustomProcessor.java:107) ... at com.glide.util.ParentedThread.run(ParentedThread.java:51) MID Server Synchronised Files Various tables extend the ecc_agent_sync_file table, and get synchronised to the MID Server in a similar way. More on that can be read in: KB0852276 How MID Server File Synchronisation works - Troubleshooting guide

These are fetched via the MIDServerFileProvider scripted SOAP service, and so the errors are only likely to have a different table name.

Script Files (ecc_agent_script_file) The following features, and possibly more, use SNMP MIB files:

Discovery Service Mapping Orchestration Integration Hub These are Powershell, Shell scripts, and other script files executed from the MID Server host or remotely.

Example logs:

2024-06-07 02:07:49 (374) API_INT-thread-3 CB57BA8F3BE2CE10E9B6368AA4E45A71 txid=0e258f433ba6 LocalhostTransactionLogger *** Start #220922 /MIDServerFileProvider.do , type: soap, user: X ... 2024-06-07 02:07:49 (385) SOAPProcessorThreadca25cbcb3ba6ce10e9b6368aa4e45a4b CB57BA8F3BE2CE10E9B6368AA4E45A71 txid=c22587cb3ba6 SignatureUtil DEBUG: Could not find the KMF Signature record using fields: document_id : 5f4015e043032110c42154249ab8f206 table_name : ecc_agent_script_file purpose : mid_script 2024-06-07 02:07:49 (393) SOAPProcessorThreadca25cbcb3ba6ce10e9b6368aa4e45a4b CB57BA8F3BE2CE10E9B6368AA4E45A71 txid=c22587cb3ba6 JustInTimeLoadingEngine SEVERE *** ERROR *** Cannot just-in-time load the signature record: PluginId is not found or unregistered 2024-06-07 02:07:49 (393) SOAPProcessorThreadca25cbcb3ba6ce10e9b6368aa4e45a4b CB57BA8F3BE2CE10E9B6368AA4E45A71 txid=c22587cb3ba6 SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, 5f4015e043032110c42154249ab8f206, for the table, ecc_agent_script_file . com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: PluginId is not found or unregistered at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordGRUsingFields(CodeSigningOutputUtility.java:120) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningSignatureRecordForRestEndPoint(CodeSigningOutputUtility.java:85) at com.snc.core_automation.security.validation.application.SignatureUtil.loadSignatureRecord(SignatureUtil.java:115) at com.snc.core_automation.security.validation.application.SignatureUtil.getSignatureV2AsMap(SignatureUtil.java:80) at com.snc.core_automation.security.validation.application.SignatureUtil.getVersionedSignatureAsJSON(SignatureUtil.java:136) at com.snc.core_automation.security.validation.application.SignatureUtilJS.jsFunction_getVersionedSignature(SignatureUtilJS.java:53) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:151) at org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:717) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:654) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2973) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1720) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:1030) at org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:161) at com.glide.caller.gen.sys_web_service_6c94a96683201000dada83ec37d929c8_script.call(Unknown Source) at com.glide.script.ScriptCaller.call(ScriptCaller.java:22) at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:160) at org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:3058) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2981) at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1720) at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:1030) at org.mozilla.javascript.InterpretedFunction.lambda$call$0(InterpretedFunction.java:161) at com.glide.caller.gen.sys_web_service_6c94a96683201000dada83ec37d929c8_script.call(Unknown Source) ... at com.glide.processors.soap.SOAPProcessorThread.run0(SOAPProcessorThread.java:131) at com.glide.util.ParentedThread.run(ParentedThread.java:51) MIB Files (ecc_agent_mib) The following features, and possibly more, use SNMP MIB files:

Discovery Service Mapping Event Management Collectors A MID Server will synchronise all MIB files from the instance on first startup, and then add/delete/update the files individually from then on.

SOAPProcessorThread... SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, 81dfdb576f820200b1099065bd3ee4a2, for the table, ecc_agent_mib. com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: 81dfdb576f820200b1099065bd3ee4a2 with signatureRecordSysId: 2fc01f7738384b7e91a7bff5b94df8c4 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) ... ACC Plugins (sn_agent_asset) The following features, and possibly more, use ACC Plugins, also called Sunsu Assets:

Agent Client Collector Framework Any app running checks in Agents, including: Agent Client Collector Monitoring Agent Client Collector for Visibility etc. Any app doing integrations via the ACC Spoke in IntegrationHub flows These are all synched to the MID Server when the MID Server Web Server and ACC Endpoint extensions are enabled, or when changes are made to individual records in the instance, perhaps when ACC apps are installed/upgraded, or custom monitoring checks are added.

SOAPProcessorThread... SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, afaa52edc32f7150f640f2ff050131c8, for the table, sn_agent_asset. com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: afaa52edc32f7150f640f2ff050131c8 with signatureRecordSysId: be8c0c7a16a74e3c9e2d699e54b1b690 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) DEBUG (FileSync:sn_agent_asset) [Events:114] Dispatching event: 'IssueEvent{operation=RESOLVE, source=CodeValidation, message=Attachment failed code signing validation, table: sn_agent_asset, attachment sys_id: 1694820a97178a503690998de053af92}' to 41 registered listeners JAR Files (ecc_agent_jar) The following features, and possibly more, use JAR Files:

External Credential Store various JavascriptProbe integrations These add extra Java Classes to the MID Server application.

SOAPProcessorThread... SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, bb3d1f433ca68210b1b3fd8eb2644ca6, for the table, sn_agent_asset. com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: bb3d1f433ca68210b1b3fd8eb2644ca6 with signatureRecordSysId: 16b5e7ab6a3d44b8849161d672273e91 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) Discovery Pattern Uploaded File (sa_uploaded_file) SOAPProcessorThread... SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, 24a729521b0eec1068cb62ca234bcb90, for the table, sa_uploaded_file. com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: 24a729521b0eec1068cb62ca234bcb90 with signatureRecordSysId: 3b18ed3bd98947e0a1aecd0d22f71944 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) ACC Config file (sn_agent_configuration_file) SOAPProcessorThread... SignatureUtil SEVERE *** ERROR *** Unable to find the signature with the sys id, cef0be841b067d10160c9602b24bcb7a, for the table, sn_agent_configuration_file. com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: cef0be841b067d10160c9602b24bcb7a with signatureRecordSysId: bb9b477e223545899d0f667046b8a991 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) When the Table is not mentioned, and other Non-MID Server-related errors These are out of scope of this KB article, but may also be seen:

Restricted Caller Access An example with sys_restricted_caller_access table name:

2024-06-07 02:48:25 (313) Default-thread-10 AE5E0B4387A642503D1C0FAF8BBB359E txid=fc7e8b032b61 SEVERE *** ERROR *** Could not find the KMF Signature record using fields: document_id : 46bcb89987bdc9103d1c0faf8bbb35bc table_name : sys_restricted_caller_access purpose : access_control_evaluation com.glide.codesigning.exception.CodeSigningException: Could not find the KMF Signature record using fields: document_id : 46bcb89987bdc9103d1c0faf8bbb35bc table_name : sys_restricted_caller_access purpose : access_control_evaluation at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:68) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordGRUsingFields(CodeSigningOutputUtility.java:84) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordFromUniqueIndex(CodeSigningOutputUtility.java:59) at com.glide.codesigning.utility.CodeSigningUtility.verify(CodeSigningUtility.java:295) at com.glide.codesigning.service.CodeSigningGlideTrustService.isTrusted(CodeSigningGlideTrustService.java:48) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccessLoader.isGlobalRCARecordInvalid(RestrictedCallerAccessLoader.java:113) ... An example without table name:

2024-06-07 02:24:26 (651) worker.1 worker.1 txid=e1f8438b1b26 SSI_3e4bdda81bacbc10bd0686ef7b4bcbca SEVERE *** ERROR *** Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: 78e72209dbfa341074243852f396192d with signatureRecordSysId: 59687f4ae8a0470790f69555343c22e1 com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: 78e72209dbfa341074243852f396192d with signatureRecordSysId: 59687f4ae8a0470790f69555343c22e1 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordGRUsingFields(CodeSigningOutputUtility.java:120) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordFromUniqueIndex(CodeSigningOutputUtility.java:63) at com.glide.codesigning.utility.CodeSigningUtility.verify(CodeSigningUtility.java:371) at com.glide.codesigning.service.CodeSigningGlideTrustService.isTrusted(CodeSigningGlideTrustService.java:48) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccessLoader.isGlobalRCARecordInvalid(RestrictedCallerAccessLoader.java:113) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccessLoader.loadAccessRecordsFromDBForScope(RestrictedCallerAccessLoader.java:65) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccessLoader.getPrivileges(RestrictedCallerAccessLoader.java:43) at com.glide.sys.fencing.cross_scope_access.CrossScopeProviderBase.lambda$getCacheEntries$0(CrossScopeProviderBase.java:49) at com.glide.sys.cache.TypeSafeCacheManager.get(TypeSafeCacheManager.java:125) at com.glide.sys.fencing.cross_scope_access.CrossScopeProviderBase.getCacheEntries(CrossScopeProviderBase.java:49) at com.glide.sys.fencing.cross_scope_access.CrossScopeProviderBase.load(CrossScopeProviderBase.java:59) at com.glide.sys.fencing.restricted_caller_access.CallerAccessRestrictionProvider.getAccessRecordsFromDescriptor(CallerAccessRestrictionProvider.java:36) at com.glide.sys.fencing.cross_scope_access.CrossScopeProviderBase.getAccessRecords(CrossScopeProviderBase.java:45) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccess.requestAccess(RestrictedCallerAccess.java:49) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccessHandler.getScopeAccessStatus(RestrictedCallerAccessHandler.java:75) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccessHandler.handleCrossScopeTableAccess(RestrictedCallerAccessHandler.java:28) at com.glide.sys.fencing.restricted_caller_access.RestrictedCallerAccessEngine.hasAccessToTableForOperation(RestrictedCallerAccessEngine.java:34) at com.glide.sys.fencing.cross_scope_access.CrossScopeAccessHandler.checkCrossScopeAccessForTable(CrossScopeAccessHandler.java:42) at com.glide.script.fencing.access.ScopedTableAccess.hasRightsTo(ScopedTableAccess.java:92) at com.glide.script.fencing.access.ScopedTableAccess.hasRightsToTable(ScopedTableAccess.java:39) at com.glide.script.fencing.ScopedGlideRecord.getScopeAccessStatusForTable(ScopedGlideRecord.java:266) at com.glide.script.fencing.ScopedGlideRecord.checkOperationPermitted(ScopedGlideRecord.java:251) at com.glide.script.fencing.ScopedGlideRecord.jsFunction_query(ScopedGlideRecord.java:188) ... at com.glide.caller.gen.sys_script_include_3e4bdda81bacbc10bd0686ef7b4bcbca_script.call(Unknown Source) Flow Engine 2024-06-07 02:15:32 (039) worker.4 worker.4 txid=6dd64f0f53aa CodeSigningUtility SEVERE *** ERROR *** Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: b4b1f78353c110107109ddeeff7b12d0 with signatureRecordSysId: 1210499b4c514639aedf86a03e4c0511 com.glide.codesigning.exception.CodeSigningException: Cannot just-in-time load the signature record: Couldn't find pluginId for documentId: b4b1f78353c110107109ddeeff7b12d0 with signatureRecordSysId: 1210499b4c514639aedf86a03e4c0511 at com.glide.codesigning.output.JustInTimeLoadingEngine.loadSignatureRecordJustInTime(JustInTimeLoadingEngine.java:89) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordGRUsingFields(CodeSigningOutputUtility.java:120) at com.glide.codesigning.output.CodeSigningOutputUtility.getCodeSigningRecordFromUniqueIndex(CodeSigningOutputUtility.java:63) at com.glide.codesigning.utility.CodeSigningUtility.verify(CodeSigningUtility.java:371) at com.glide.codesigning.service.CodeSigningGlideTrustService.isTrusted(CodeSigningGlideTrustService.java:48) at com.glide.CodeSigning.isTrusted(CodeSigning.java:11) at com.glide.flow.compiler.component_block.ActionBlockCompilerBase.validateStepInstance(ActionBlockCompilerBase.java:111) at com.glide.flow.compiler.component_block.ActionBlockCompilerBase.validateAction(ActionBlockCompilerBase.java:104) at com.glide.flow.compiler.component_block.ActionBlockCompiler.createAction(ActionBlockCompiler.java:76) at com.glide.flow.compiler.component_block.ActionBlockCompiler.build(ActionBlockCompiler.java:60) at com.glide.flow.compiler.component_block.ActionBlockCompiler.build(ActionBlockCompiler.java:55) at com.glide.flow.compiler.FlowBlockCompiler.buildFlowBlock(FlowBlockCompiler.java:70) at com.glide.flow.compiler.logic_block.TryCompiler.build(TryCompiler.java:33) at com.glide.flow.compiler.component_block.FlowLogicBlockCompiler.addFlowLogic(FlowLogicBlockCompiler.java:59) at com.glide.flow.compiler.component_block.FlowLogicBlockCompiler.build(FlowLogicBlockCompiler.java:39) at com.glide.flow.compiler.FlowBlockCompiler.buildFlowBlock(FlowBlockCompiler.java:75) at com.glide.flow.compiler.FlowGlideCompiler.compile(FlowGlideCompiler.java:108) at com.glide.flow.compiler.v2.EngineVersionAwareFlowCompiler.compile(EngineVersionAwareFlowCompiler.java:52) at com.glide.flow.providers.FlowGlideProvider.getFlowPlan(FlowGlideProvider.java:1887) at com.glide.flow_trigger.engine.FlowPlanRetriever.recompileAndCacheProcessPlan(FlowPlanRetriever.java:200) at com.glide.flow_trigger.engine.FlowPlanRetriever.compile(FlowPlanRetriever.java:304) at com.glide.plan.runners.FlowObjectAPI.compilePlan(FlowObjectAPI.java:339) at com.glide.plan.runners.scriptable.ScriptableFlowRunner.jsFunction_compile(ScriptableFlowRunner.java:697) ... at com.glide.caller.gen.sys_trigger_dde1984a0f1330102841244ac4767e5a.call(Unknown Source) ... Solution This error message shouldn't happen, but can be ignored in relation to the MID Server if Code Signing is not enabled in the instance.

Script signatures should always be created out-of-box on the instance, because Code Signing might be activated. Development teams of the features that run in MID Servers have been working on this and have been adding the signatures to out of box records and tables.

The Xanadu release should fix this, by adding most of these signatures to the build system, but others are targeted for Yokohama release. All ecc_agent_... tables, and those listed above that extend ecc_agent_sync_file should be fixed in Xanadu (STRY57271154).

A Xanadu Patch 1 demo instance, with most of the ITOM/ITOM apps installed, includes sn_kmf_record_signature records for these tables and number of records for each. You would expect something similar in this list, depending on what's installed:

https:// .service-now.com/sn_kmf_record_signature_list.do?sysparm_query=GROUPBYtable_name ecc_agent_mib (1920) ecc_agent_script_file (3302) ecc_agent_script_include (11859) mid_server_tools (30) sa_pattern (6249) sa_uploaded_file (47) sys_data_source (374) sys_hub_action_type_definition (232) sys_hub_flow (124) sys_hub_input_scripts (7949) sys_hub_pill_compound (17187) sys_hub_step_instance (11993) sys_kmf_crypto_caller_policy (1049) sys_kmf_crypto_module (631) sys_restricted_caller_access (1055) sys_rest_message (55) sys_rest_message_fn (93) sys_rest_message_fn_param_defs (34) sys_rest_message_headers (68) sys_script (94978) sys_script_include (67541) sys_soap_message (51) sys_soap_message_function (782) sys_soap_message_parameters (1036)

The out-of-box records, that are part of apps and plugins, in these tables, would have signatures created at build time. Extra records added to these tables by customers would not be expected to have signatures automatically.

System Exports Sets (ecc_agent_attachment) System Exports Sets place files in the agent\export folder of the MID Server install folder. These files as created in the instance, based on the Export Set definition in sys_export_set, and placed temporarily in the ecc_agent_attachment table. All the MID Server then does is run a StreamPipeline probe to get that record from the instance and copy it to disk, but that does also use SignedGlideRecord, and so does expect Signatures to be available if needed.

As of Yokohama, these records are not signed. This would require something to sign the records at runtime, as they are created.

PRB1989657 / KB2763649 System Export Sets don't work on Code Signing enabled instances For instances without code signing enabled, after PRB1892912 is fixed in Australia, errors won't be seen in agent logs, and MID Server issues records won't be created for these Export Set attachments.

Agent Client Collector (sn_agent_asset, sn_agent_configuration_file) Agent Client Collector based apps cannot currently be used with Instances that have Code Signing enabled.

No signature records files are generated for any sn_agent_asset.* or sn_agent_configuration_file.* records, which will prevent those being synched to MID Servers.

PRB1770274/KB1707055 ACC Plugins (Assets) cannot be synced to MID Servers if Circle of Trust (code signing) is enabled in the instance, breaking all Agent Client Collector related features That depends first on the following being completed: PRB1826273 Code Signing KMF Signature Entries missing. being tracked internally as STRY59015339, which is complete in Zurich. Since Zurich, where the tables are now capable of having signatures, each individual ACC-based App would need to be updated to include signatures for those app's own records.

As of these versions, Signatures are not yet included:

Agent Client Collector for Investigation [sn_acc_adapter] v? Agent Client Collector for Visibility Content [sn_acc_vis_content] v? Agent Client Collector for Visibility [sn_acc_visibility] - Probably obsolete, and replaced by sn_acc_vis_content scope copies. Agent Client Collector Framework [sn_agent] v6.0.1 (PRB1770274) Agent Client Collector Monitoring [sn_itmon] v? Kubernetes Visibility Agent 3.10.2 Live CI View [sn_live_ci_view] v? Remediate Zscaler VPN [sn_acc_zscaler] v? ITSM Virtual Agent Conversations [sn_itsm_va] v? Synthetic Monitoring [?] v? (PRB1952528) DEX Content Playbook [sn_dex_content] v? Cloud Native Operations (aka Kubernetes Visibility Agent) [sn_k8s] v? Discovery Aspects of Discovery are not fully compatible when Code Signing is enabled on an instance. This is due to the fact that setting the ECC firewall configuration to allow all Discovery to be accepted allows a Quick Discovery for one server to succeed. Without this change, the default ECC firewall rule is to reject all ECC queue messages unless they contain valid signatures. Some sensor messages in the ECC queue do not have a valid signature

PRB1811227/KB1703730 Discovery Sensors and Probes do not support Code Signing being tracked internally as STRY58742389, which is currently still pending. IntegrationHub When enabling Full Validation of Flows (com.glide.hub.code_signing.full.validation.enabled) many Flow/IntegrationHub records are still missing required signatures.

In the words of the scan check in the ISM Monitoring app: "The com.glide.hub.code_signing.full.validation.enabled property controls the level of strictness of checks enforced when Code Signing is fully enabled. When the property is set to "true" stricter checks are enforced and certain Flow Designer features incompatible with Code Signing are blocked from executing. If the property is insecurely set to "false" or unset, legacy checks are performed instead and a subset of the Code Signing-incompatible Flow Designer features are allowed to execute."

Synthetic Monitoring In Synthetic Monitoring 1.3.1, there are currently 2 MID Server Script Includes:

SyntheticMonitoringCheckRunner is the probe for checks via MID Server locations. SnytheticMonitoringHTTPResultParser is the mid server side component of the ACC check type, for checks via ACC Related problem:

PRB1952528 Code Signing signatures are missing for Synthetic Monitoring MID Server Script Includes