Edge Proxy Authentication Fails With Warning: Received an error response code Signed certificate in response to 500 from the ServiceNow instance : "instance1.service-now.com". Message: java.security.KeyStoreException: Uninitialized keystore | Detail: - Support and Troubleshooting

Edge Proxy Authentication Fails With Warning: Received an error response code Signed certificate in response to 500 from the ServiceNow instance : "instance1.service-now.com". Message: java.security.KeyStoreException: Uninitialized keystore | Detail: Issue  Using the Madrid release or above, an Edge Proxy is brought online and the "Authentication" is set to "Unauthenticated".

The proxy is selected and you select the "Authenticate" button, but the proxy does not move from "Unauthenticated" to "Authenticated".

Checking the Edge Proxy /logs/edgeencryption.log shows this during the Authentication attempt:

2019-07-12 07:44:34,298 WARN Received an error response code Signed certificate in response to 500 from the ServiceNow instance : "instance1.service-now.com". Message: java.security.KeyStoreException: Uninitialized keystore | Detail:

Checking the instance node logs for the same time will show messages like this:

2019-07-12 04:44:32 (253) http-32 New transaction 0CF7839ADB6A3740D5CFF2131F9619C9 #21221 /api/sn_edge_encryption/v1/sync/authenticate 2019-07-12 04:44:32 (271) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a #21221 /api/sn_edge_encryption/v1/sync/authenticate Parameters ------------------------- edge_encryption_pool= api=api message=O774turqrWPROsh7Lk3Z15jKvrjw5WTs0JlQhSe3KHFd0qcddaF1W8s2NN1R098dkOdBDP21yrpWnZ8z9D8QrGyHlNKT5Rcx7iCt4WJoy94TLTZRflNSZ0qg4Vc3ZVPitEYdcckDaK9BhSyLbALbJC2idDm4afmKlCE-Vct79Wsnllw9HDT7eYUXMxN5If5505Zay6o6ygoSlA6osmEmnej7nw1wCH2MQrmcHTItkrXxJfrLZCEIhQD6OQrSc_c5Ut5A4SB9iSClx_FqZVCBtXuPqdr2CcDgPNwAcy6CIoKoeMSkWKUNPb0DLK44nxNvRsOTlB-7mhpWLppTOoic-20lCO7QRxu8FYcU5ZlideUqiY7ki2hekw7HEnTkWyRkvwbUbIfRW69hXChpmHYj0xsUlfXBDVlWiGyCo_fCO8emdy4aRvPh0A6CttNIkleB1Rixz4O9LcPfd-jbenowgJ4YKoR3U2oKxl_vV_DJziyj5wtuctcLFpkrWYTnZl_b hash= 2019-07-12 04:44:32 (271) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a *** Start #21221 /api/sn_edge_encryption/v1/sync/authenticate, user: edge 2019-07-12 04:44:32 (348) http-39 New transaction 3267039ADB6A3740D5CFF2131F9619C7 #21222 send /amb/meta/disconnect 2019-07-12 04:44:32 (394) AMB_SEND-thread-36 3267039ADB6A3740D5CFF2131F9619C7 txid=2d08879adb6a *** Start #21222 send /amb/meta/disconnect, user: a.b@snc 2019-07-12 04:44:32 (394) AMB_SEND-thread-36 3267039ADB6A3740D5CFF2131F9619C7 txid=2d08879adb6a [AMB] AMBSessionManager cometd_session_id=5315xv7xd1qvzwd1ci3ozv0pzlxm Session removed 2019-07-12 04:44:32 (394) AMB_SEND-thread-36 3267039ADB6A3740D5CFF2131F9619C7 txid=2d08879adb6a *** End #21222 send /amb/meta/disconnect, user: a.b@snc, total time: 0:00:00.046, processing time: 0:00:00.046, SQL time: 0:00:00.001 (count: 1), source: 199.91.140.61 2019-07-12 04:44:32 (411) http-36 New transaction 3267039ADB6A3740D5CFF2131F9619C7 #21223 /api/now/ui/date_time/legacy 2019-07-12 04:44:32 (422) Default-thread-206 SYSTEM txid=35084b9adb6a WARNING *** WARNING *** Cookie token not in database: SCv3:l9JfJYLGUJ9wq8VExfxycslGKCGsvhQx:ysQo0cuHrP3x8dj3YJFcXChcqW1NePzt5+D5JOPy3yU= 2019-07-12 04:44:32 (422) Default-thread-206 SYSTEM txid=35084b9adb6a WARNING *** WARNING *** aborted activity cookie update: U0N2MzpsOUpmSllMR1VKOXdxOFZFeGZ4eWNzbEdLQ0dzdmhReDp5c1FvMGN1SHJQM3g4ZGozWUpGY1hDaGNxVzFOZVB6dDUrRDVKT1B5M3lVPQ== 2019-07-12 04:44:32 (423) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a Bypassing ACL checks for a public page: /api/now/ui/date_time/legacy 2019-07-12 04:44:32 (423) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a #21223 /api/now/ui/date_time/legacy Parameters ------------------------- api=api 2019-07-12 04:44:32 (423) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a *** Start #21223 /api/now/ui/date_time/legacy, user: a.b@snc 2019-07-12 04:44:32 (427) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a *** End #21223 /api/now/ui/date_time/legacy, user: a.b@snc, total time: 0:00:00.014, processing time: 0:00:00.014, SQL time: 0:00:00.001 (count: 4), source: 199.91.140.61 , type:rest, method:GET, api_name:now/ui, resource:now/ui/date_time/legacy, version:Default, user_id:a.b@snc, response_status:200 2019-07-12 04:44:32 (523) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** string may not be encrypted : Given final block not properly padded. Such issues can arise if a bad key is used during decryption. 2019-07-12 04:44:32 (524) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a SEVERE *** ERROR *** Attachment is missing for certificate sn_edge_instance_selfcert - 8d80f1a6dbcc3b409063f4eabf961922 2019-07-12 04:44:32 (528) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** Evaluator: java.security.KeyStoreException: Uninitialized keystore Caused by error in sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script at line 22 java.security.KeyStore.getKey(KeyStore.java:1021) com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.fetchSigningKey(AuthenticationProcessorAPI.java:393) com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.jsStaticFunction_generateSignedCertificate(AuthenticationProcessorAPI.java:184) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138) org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670) org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582) org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_process_1(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:22) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_script_0(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:1) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563) org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3429) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.exec(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279) com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118) com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82) com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309) com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214) com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201) com.glide.rest.service.custom.CustomService.runScript(CustomService.java:96) com.glide.rest.service.custom.CustomService.execute(CustomService.java:83) com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:37) com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:290) com.glide.processors.AProcessor.runProcessor(AProcessor.java:531) com.glide.processors.AProcessor.processTransaction(AProcessor.java:229) com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:188) com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:177) com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31) com.glide.sys.Transaction.run(Transaction.java:2147) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

2019-07-12 04:44:32 (528) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** Evaluator: java.security.KeyStoreException: Uninitialized keystore Caused by error in sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script at line 1 java.security.KeyStore.getKey(KeyStore.java:1021) com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.fetchSigningKey(AuthenticationProcessorAPI.java:393) com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.jsStaticFunction_generateSignedCertificate(AuthenticationProcessorAPI.java:184) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138) org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670) org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582) org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_process_1(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:22) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_script_0(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:1) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563) org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3429) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.exec(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279) com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118) com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82) com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309) com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214) com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201) com.glide.rest.service.custom.CustomService.runScript(CustomService.java:96) com.glide.rest.service.custom.CustomService.execute(CustomService.java:83) com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:37) com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:290) com.glide.processors.AProcessor.runProcessor(AProcessor.java:531) com.glide.processors.AProcessor.processTransaction(AProcessor.java:229) com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:188) com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:177) com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31) com.glide.sys.Transaction.run(Transaction.java:2147) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

2019-07-12 04:44:32 (530) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a SEVERE *** ERROR *** java.security.KeyStoreException: Uninitialized keystore com.glide.rest.domain.ServiceException: java.security.KeyStoreException: Uninitialized keystore at com.glide.rest.service.custom.CustomServiceExceptionResolver.throwServiceException(CustomServiceExceptionResolver.java:82) at com.glide.rest.service.custom.CustomServiceExceptionResolver.throwServiceException(CustomServiceExceptionResolver.java:77) at com.glide.rest.service.custom.CustomServiceExceptionResolver.resolveForException(CustomServiceExceptionResolver.java:39) at com.glide.rest.service.custom.CustomServiceResultHandler.handle(CustomServiceResultHandler.java:22) at com.glide.rest.service.custom.CustomService.execute(CustomService.java:84) at com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:37) at com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:290) at com.glide.processors.AProcessor.runProcessor(AProcessor.java:531) at com.glide.processors.AProcessor.processTransaction(AProcessor.java:229) at com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:188) at com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:177) at com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31) at com.glide.sys.Transaction.run(Transaction.java:2147) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.mozilla.javascript.JavaScriptException: java.security.KeyStoreException: Uninitialized keystore at org.mozilla.javascript.Context.makeJavaScriptException(Context.java:1935) at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1921) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:143) at org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670) at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_process_1(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:22) at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) at org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651) at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_script_0(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:1) at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3429) at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.exec(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script) at com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279) at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118) at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82) at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309) at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214) at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201) at com.glide.rest.service.custom.CustomService.runScript(CustomService.java:96) at com.glide.rest.service.custom.CustomService.execute(CustomService.java:83) ... 11 more

2019-07-12 04:44:32 (531) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** #21221 [REST API] RESTAPIProcessor : Handling exception java.security.KeyStoreException: Uninitialized keystore 2019-07-12 04:44:32 (532) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a *** End #21221 /api/sn_edge_encryption/v1/sync/authenticate, user: edge, total time: 0:00:00.276, processing time: 0:00:00.276, SQL time: 0:00:00.009 (count: 20), source: 199.91.140.61 , type:rest, method:GET, api_name:sn_edge_encryption/sync, resource:sn_edge_encryption/v1/sync/authenticate, version:v1, user_id:4fd4b095dbdcb200d5cff2131f961928, response_status:500

Release Starting in the Madrid Release, the issue will most likely be seen on instances that have been restored from a backup or following a clone.

Cause At System Definition -> Certificates there may be one or multiple Certificates named "sn_edge_instance_selfcert"

The certificates(s) will either not have an attachment named "sn_edge_instance_selfcert.p12" or that attachment may be from a different instance due to cloning or restore from backup.

The missing attachment or incorrect attachment will cause the proxy not to be able to Authenticate.

From the node log above you will see reference to the "sn_edge_instance_selfcert" in the error messaging, in this case the attachment was missing in the certificate:

SEVERE *** ERROR *** Attachment is missing for certificate sn_edge_instance_selfcert - 8d80f1a6dbcc3b409063f4eabf961922

Resolution (1) Go to System Definition -> Certificates

(2) Delete all certificates there with the name "sn_edge_instance_selfcert"

(3) Go back to the Edge Encryption Configuration -> Proxies -> All -> select the proxy to be Authenticated and select "Authenticate" if there are no other issues the proxy should now go to "Authenticated". A new corrected "sn_edge_instance_selfcert" certificate will be created with the proper attachment.

(4) To cover the clone case for future clones: For what Edge Encryption related sys_certificate records to exclude during a clone see KB0696135