Handle issues with integration runs Product Success Playbook Handle issues with integration runs A step-by-step guide to analyze and remediate failed integration runs Table of Contents Summary Goal of this Playbook Audience Problem Overview Executive Summary How this playbook can help you achieve business goals How this playbook is structured Problem Analysis Upstream Causes Downstream Consequences Impact on Your Business Engagement Questions Remediation Plays Summary Play 1: Review your data Play 2: Fix Play: Troubleshoot errors based on failure notes Play 3: Fix Play – Troubleshoot common issues Data Governance Summary Goal of this Playbook Identify and rectify issues with Vulnerability integrations to help enrich the vulnerability data on your instance Details about this playbook Author Bibu Elias Punnachalil Reviewer Siva Reddy Mallu Date 12/30/2022 Addresses HSD # HSD0902655 Applicable ServiceNow Releases All Releases Prerequisites Time Required Approximately 1 to 4 hours (depending on your environment) Audience Vulnerability Administrator, Vulnerability Analysts, Remediation teamsServiceNow Administrator Problem Overview The Vulnerability Response (VR) application can have millions of vulnerabilities imported using integration runs with third-party Vulnerability Assessment (VA) applications, such as Qualys, Tenable, and Rapid7. Integration run failures adversely affect ServiceNow ingestion of vulnerable items (VI) and vulnerability remediation Executive Summary How this playbook can help you achieve business goals This playbook addresses the need to ingest Vulnerable Item records in ServiceNow to get the best value from a VR implementation. This will help to get more value out of the VR implementation How this playbook is structured Play 1 (a review data play) helps you review the failed integration runsPlay 2 (remediation) provides remediation steps required to fix the typical integration run errorsPlay 3 (remediation) provides remediation step to increase the probability of successful integration runsPlay 4 (a Data Governance play) lists the guidelines and processes for ensuring successful integration runs Problem Analysis Upstream Causes Failures due to version/change in ServiceNow platform or third partyIntegration configuration is not optimally setupIntegration runs/dashboard are not monitored Downstream Consequences Data Consequence Missing vulnerable itemsStale vulnerable items Operation Consequence Incomplete response to vulnerabilitiesFrustration, lack of confidence in the VR implementation App Consequence Dashboards & reports using VI Status information for vulnerability response analysis will be of limited use Impact on Your Business More stale VIs active will negatively impact the effectiveness and efficiency of your Vulnerability Remediation teams, vulnerability teams and audits. Also, the missing VI records may delay response to the associated vulnerabilities Security MTTR Missed vulnerability identification.Focus on stale vulnerable detections impacts team productivitySlower response to vulnerability remediation Audit/Compliance Incomplete VI data Engagement Questions: Consider the answers to these questions: Are you on Vulnerability Response V14.0 or above?Did you upgrade?Did the VR implementation follow the "crawl-walk-run" approach? Remediation Plays SummaryThe table below lists and summarizes each of the remediation plays in the playbook. Details are included later Play Name Play 1: Review your data What this play is about List failed integration runs Required tasks Follow the steps provided in the play Play 2: Fix typical errors appearing the notes field What this play is about Troubleshoot Vulnerability Integration Run errors based on the content in the “notes” field Required tasks Check the table for matching error and follow the associated solution suggestion Play 3: Troubleshoot integration run failures What this play is about Troubleshoot the typical issues in integration run issues Required tasks Check the table for matching issue and follow the associated solution suggestion Data Governance What this play is about Monitor and maintain successful integration runs Required tasks Follow the steps provided in the play Play 1 - Review your data What this Play is about List the failed integration runs from all third-party schedules Required tasks Go to navigator and type sn_vul_integration_run.listChange the filter with condition substate!=successSort by the field Updated in descending order Check the notes field to find the reason for the failure Play 2 – Fix typical errors from the failed run notes What this Play is about Analyze typical integration run errors and how to fix them Required tasks Check if the reason for failure matches one of the below, if yes, apply the corresponding solution Notes - Error String Potential Solution Encountered error running the integration. Error: Invalid response code received 1) Check if there is any password change on the third-party tool2) Check if the third-party tool is providing the response using REST interface. Troubleshoot the third party tool or network to check why the response is not being sent Encountered error running the integration. Error: Cannot run REST based integration without **** credentials specified Navigate to "Vulnerability Response > Administration > Setup assistant", then click on "Scanner integrations" at the bottom and click on edit in Qualys. Configure the credentials and other integration level settings. The CI lookup reapply job is still in progress. You can create a new integration run after it is complete. 1) Navigate to System Scheduler >Scheduled Jobs > Scheduled Jobs 2) Filter for the job with name "Reapply CI lookup rules"3) Check the state of the job to see if it is still running Encountered error running the integration. Internal Error: String object would exceed maximum permitted size of 33554432 1) The platform has a hardcoded value of 32 MB for the String object, and it is not customizable via any system property.2) Investigate the data sent by the third party tool Encountered error running the integration. Error: Invalid response code received from Get Plugins: Unauthorized 401 The URL is incorrect. 1) Sometimes the plugin is not activated in some situations like cloning etc. Validate if the plugin is installed & active 2) Enable disabled plugin by going to the related links in the plugin and click on Activate/Repair Play 3 - Fix Play – Troubleshoot integration run failures What this Play is about This play helps in some of the steps which can be taken for troubleshooting integration run failures Required tasks The table below may assist troubleshooting some integration run issues Issue Potential Solution After clicking on the Execute Now button if Integration Run is not created: Navigate to Vulnerability Response > Administration > Integrations. Check whether the Integration is active. Vulnerabilities integration run fails with 429 error. a) You cannot create multiple export jobs on a single account while the export job status is processing.b) Wait for the previous export status job to finish, then trigger the new integration run. Type “sn_vul_integration_run.list” in the navigator to list all the integration runs. Check if the job is still executing Vulnerabilities / assets integration getting timed out. Export status job (2nd integration process) times out, please check the value of export timeout in Integration Instance and increase the value accordingly. In the application filter navigator, type “integration instances”. Select the Integration Instance showing the error. In the integration instance, check for the Integration Instance Parameters tab and increase value for the configuration export_timeout.In case parameter does not exist create a new parameter for this. Vulnerabilities integration not creating the fixed vulnerabilities. Check whether the insert fixed option is enabled in integration instance. In navigator Type “integration instances”. Select the Integration Instance showing the error. In the integration instance, check for the Integration Instance Parameters tab and for the value of the configuration “insert_fixed” to true. In case parameter does not exist create a new parameter for this. Integration run fails. Check the integration run notes and system logs and take actions accordingly. Type “sn_vul_integration_run.list” in the navigator to list all the integration runs. Open the failed job – check the notes tab for the message and go to Vulnerability Integration logs to see if any log messages Vulnerabilities integration is not fetching all vulnerabilities. Check the query filter configured for the integration instance (1) Navigate to Vulnerability Response > Administration > Setup Assistant (2) Under "Integration Configuration" select "Scanner Integrations" (3) Under tenable find The integrations. Click Edit (4) Select option 4 -> Vulnerabilities Import Configuration 5) Configure the filters Integration fails with "Invalid MID server configuration" error. Check whether the configured MID server is up and validated correctly. Using the application filter navigator, navigate to MID Servers > Servers and check the status of the MID Server. If status is found to be down, troubleshoot & fix the issue Integration process fails with message "No Response received for 5 mins". a) Check the ecc_queue table whether an input record is received for the integration process you are looking for. b) If there is no response from Tenable.sc the process will be marked as retry. c) The integration process will be retried 5 times and marks the integration run status as failed. d) Troubleshoot the connection to Tenable by navigating to Vulnerability Response > Administration > Setup Assistant, click Scanner Integrations Data Governance What this Play is about Monitor and maintain the remediation task rules with no assignment group Required tasks Regularly execute play 1 to review remediation tasks with no assignment groupSetup a process to review and fine tune remediation target rules on a regular basis Congratulations You have completed this Product Success Playbook.