Qualys Integration is not pulling in all the new Vulnerability data into ServiceNow. - Support and Troubleshooting

Qualys Integration is not pulling in all the new Vulnerability data into ServiceNow. Issue .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } }  Qualys Integration is not pulling in all the new Vulnerability data into ServiceNow

Problem The Qualys Pagination process is failing to pull in all nightly Vulnerabilities. The following Errors were shown in the logs:

14:44:06.918 Debug worker.1 worker.1 DEBUG: REST Msg Outbound - RESTMessageClient : Response: Outbound REST Response

HTTP Status: 401 Unauthorized

Symptoms .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Symptoms

Troubleshooting this was difficult because we found the 401 Unauthorized error to be random. Sometimes the 401 error happened right when the job started and would only pull a few records in. Other times it would run for a hour or more and then throw the error. The other strange part was the credentials were correct. This took as down an incorrect path of troubleshooting the Qualys user on the qualys server, checking to see if the account was locked or having issues. Later we were able to correlate the following errors at the time of the 401 messages:

2017-09-17 23:33:27 (244) worker.0 worker.0 WARNING *string may not be encrypted: aEo:6TqjgxlpyX0dISG/Xz7omOHOVaxgy2Q8q9D9N7I0pN4= : Input length must be multiple of 8 when decrypting with padded cipher

2017-09-17 23:33:27 (242) worker.0 worker.0 WARNING *** WARNING *** StorageEncrypter: no registered Key-Provider for hashKey: aEo 2017-09-17 23:33:27 (243) worker.0 worker.0 WARNING *** WARNING *** StorageEncrypter: no registered Key-Provider for hashKey: aEo 2017-09-17 23:33:27 (243) worker.0 worker.0 WARNING *** WARNING *** StorageEncrypter: no registered Key-Provider for hashKey: aEo 2017-09-17 23:33:27 (244) worker.0 worker.0 WARNING *** WARNING *** StorageEncrypter: no registered Key-Provider for hashKey: aEo 2017-09-17 23:33:27 (244) worker.0 worker.0 WARNING *** WARNING *** StorageEncrypter: no registered Key-Provider for hashKey: aEo 2017-09-17 23:33:27 (244) worker.0 worker.0 WARNING *** WARNING *** string may not be encrypted: aEo:6TqjgxlpyX0dISG/Xz7omOHOVaxgy2Q8q9D9N7I0pN4= : Input length must be multiple of 8 when decrypting with padded cipher 2017-09-17 23:33:27 (244) worker.0 worker.0 DEBUG: REST Msg Outbound - RESTMessageClient : Executing synchronous request 2017-09-17 23:33:27 (244) worker.0 worker.0 DEBUG: REST Msg Outbound - RESTMessageClient : Executing: Outbound REST Message/Method: sn_vul_qualys.Host Detection Pagination Message/get HTTP Request: GET https://xxxxx.xxxxxx.com/api/2.0/fo/asset/host/vm/detection/?output_format=XML&detection_updated_since=2017-09-17T06%3A07%3A03Z&show_tags=1&action=list&truncation_limit=500&status=New%2CRe-Opened%2CActive%2CFixed&severities=3%2C4%2C5&id_min=174549689 X-Requested-With: ServiceNow Authentication type: basic Authentication profile: Profile name: null Username: xxxxxxx Password: ******** Mutual Auth: false ECC Queue: false Release .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } All

Cause .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Cause

We were then able to determine that the above errors were always coming from the same node. Since all the nodes were working and would connect to Qualys without issue and only one node was having issues, we requested the Internal ServiceNow team review the differences between a working node and the failing one. They found the following:

The node was provisioned as mini and not small like the others. Then comparing it with a working node we can see that in overrides.d there are files missing, ie: [app129025.sjc101:/glide/nodes/bbitprod056_16001/conf/overrides.d]# ls -la total 24 drwxr-xr-x+ 2 p16001 p16001 4096 Apr 12 01:10 . drwxr-xr-x+ 5 p16001 p16001 4096 Sep 19 23:10 .. -rw-r-----+ 1 p16001 p16001 146 Apr 12 01:10 README [xxxxxx.xxxxxx:/glide/nodes/xxxxxxxx/conf/overrides.d]# ls -la total 40 drwxr-xr-x+ 2 p16017 p16017 4096 Feb 13 2017 . drwxr-xr-x+ 5 p16017 p16017 4096 Sep 19 23:10 .. -rw-r-----+ 1 p16017 p16017 97 Feb 13 2017 glide.properties -rw-r-----+ 1 p16017 p16017 468 Feb 13 2017 glide.safenet.properties -rw-r-----+ 1 p16017 p16017 146 Feb 13 2017 README

Resolution .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Resolution

The fix was to decommission the bad node that was set up as a Mini and Re-provision a new node as the correct size of Small . The main cause was the Mini node did not have the correct glide properties files to allow it to properly run the Storage Encryptor to encrypt/decrypt the password that was being passed from SN to the Qualys server.