Configure DevOps with Application Vulnerability Response for better visibility into CI/CD pipeline - Support and Troubleshooting

Configure DevOps with Application Vulnerability Response for better visibility into CI/CD pipeline Instructions Introduction

Application Vulnerability Response is an additional feature set in the existing Vulnerability Response store application enabling users to track and prioritize application vulnerabilities found during Dynamic Application Security Testing (DAST) or Static Application Security Testing (SAST) or Software Composition Analysis (SCA). These vulnerabilities are generally identified by third-party systems such as Veracode, Fortify, Qualys WAS, or other application vulnerability scanners. An Application Vulnerability Integration is developed using ServiceNow platform components and is responsible for importing Application Releases, Application Scan Summaries, Application Configuration Items, and Application Vulnerable Items into ServiceNow.

This guide is intended to provide developers with step-by-step instructions to create new DevOps integration to provide the end user with the visibility of Vulnerability summary info in their DevOps pipelines.

If your requirement is adding a DevOps integration on top of an existing VR integration, please skip to the section "DevOps Integration".

General flow

Integrations are recommended to utilize the existing Vulnerability Response Integration Framework, AVR Import API, and common Configuration Page. This will create a common user experience for customers using multiple scanners and require development of fewer custom components.

Your Application Vulnerability Integration will include an Integration record, which represents an interaction with the scanner API (for example, ingesting Release data or Vulnerable Item data). Executing this integration will create an Integration Run record and Integration Process record. The Run represents an execution of the Integration, and a Process represents a single interaction with the API – for example, a page of data. Creation of Runs and Processes will be handled as part the Vulnerability Response Integration Framework.

The Integration record will specify a construction script, which should reference a script include that handles interaction with the scanner API. This script include should handle error responses from the API (and retry if appropriate), save a successful response as an attachment on the corresponding Process record, and queuing subsequent Integration Process records if more calls are needed.

The Process and attachment will then be associated with a Data Source Import Queue record, and the Process record set to “waitComplete” status. The system will wait for the next available Data Source (defined as part of the Integration) and invoke the associated Transform Map. The Transform Map should define an “onComplete” script that will invoke a report processor script include.

This script include will be responsible for parsing the response. The parsed response should be placed in a map object and passed to the AVR Import API. The API is responsible for validating the provided object, performing CI Lookup and Severity Mapping, and saving the record. Once complete, the Process will be set to “Complete” status.

Once all Process record have been marked as Complete, the Run will be marked as complete, and the Integration will run again at the next scheduled interval.

When developing your Integration, you will, at minimum, need define your Configuration fields, Integration record, data sources, import templates, requesting script include, response processing script include, and transform map. All persistence of data from the response processing script include should be routed through the AVR Import API.

Creating a plugin

Overview A plugin is the top-level component of your scanner integration. It will define both the package that is downloaded via the ServiceNow store by other users, and the scope within which your code will run. All logic developed as part of the integration - scripts, business rules, transform maps, etc – will be part of the plugin.

Steps to create a plugin Open Studio and Create a new Application. At the prompt, select the “Start from Scratch” option.

In the sample code, we are using a scope of “sn_app_vul_demo” – note that custom scopes are normally prepended with “x_”. If you copy/paste any of the example code in this document, make sure you replace that scope with your own.

On the new Application page, you will need to add a dependency. Go to “File” and “Settings”, Select the Dependencies tab on the bottom of the Application screen and select “Edit”.

From here you can add “Vulnerability Response” as a Dependency by scrolling down to the bottom and clicking “Edit...” button on the Dependencies tab.

NOTE: The slush bucket displayed via the “Edit” button may not have Vulnerability Response as an available option. To work around this, instead right click the filter in the related list view and choose “Open new window”.

In the newly-displayed list view, click “New” and choose “Vulnerability Response” as a dependency. The package should default to your new app.

When developing, ensure that you are currently in your new application’s scope. In Studio, this is automatic but most Application Vulnerability Response Integration development cannot be done in Studio and is done by creating elements in the normal ServiceNow application. This guide is written assuming you are in the normal ServiceNow view of the application so navigate back to the homepage of your instance. Certain objects created in this view will be added to the current scope, so you want to make sure you are in the correct application scope. You can view and change the scope via the Application Picker. To enable the Application Picker, from the main ServiceNow page, click the settings icon in the upper right corner. Click on “Developers” options and select “Show update set picker in header”.

Once you have this selected, you can choose your current application in a drop down in ServiceNow. Select your current application and you are ready to start manually creating your configuration records in your scope. Note that you should not stay in the application’s scope when you are not developing on your application as it may create unnecessary records in your application.

Now that the plugin and scope have been established, and the application selected, you are ready to begin building the business logic of your Application Vulnerability integration.

For more details on this step, see “ Creating Applications ” in the online documentation.

Configuration page

Overview

The Vulnerability Response Integration Framework includes four tables for establishing a configuration for the integration. Configuration consists of the parent Third Party Integration record, and a definition of required Configurations for that Integration. Users can then create instances of that integration, representing a particular subscription or configuration. Creating a new Instance will execute a setup script and create associated Instance Parameters based on the Integration Config.

These configurations can be created directly on the form:

sn_sec_int_integration – Top level representation of the integration sn_sec_int_config – Defines the configuration types that the integration expects sn_sec_int_impl – Represents a single instance of the integration sn_sec_int_impl_config – Contains the configuration value for a given instance

Steps to create a configuration

First, navigate to the Integration form and create the initial Third-Party Integration definition. This should also include the script to validate the credentials that users will later enter, which will set the instance validation status.

Next, define the different types of configurations that will be need via the Integration Config related list shown on the Integration form.

Next, create a new Instance record. This should have the “default” field set to True to indicate that this is the initial instance.

Lastly, the Instance configs can be set. Be sure that unencrypted fields are saved to the “Value” column, and encrypted fields are saved to the “Password value” column.

Displaying on the UI Page

You can also show these configurations in a more user-friendly way by creating a new Module. Set the Link Type to “URL (from Arguments)”, and set the Arguments to:

ui_page.do?sys_id=1b6ed2b3b74e04102503f3fdde11a93c&integration_id=

Integration record

Overview

A "Vulnerability Integration" (sn_vul_integration) is a scheduled job that imports external data and uses that data to create and/or update records in ServiceNow. Typical VR integrations will contain several Vulnerability Integration records for different data types, endpoints, or import configurations, all of which can be scheduled independently by the end user. Not to be confused with the "Third Party Integration" (sn_sec_int_integration) table. VR integrations will only have one Third Party Integration record, which is used to define the vendor name (and integration type) within the list of VR vendor integrations.

Vulnerability Integrations do their imports in two steps. First, there’s the retrieval step, where the integration fetches the data from the external source. Then, there’s the processing step, where the data is transformed into the correct format and records are created/updated in ServiceNow. The retrieval process is managed within the “Integration script”. Processing can be done directly with Transform Maps, or it be done inside a “report processor” script. These script types are discussed in their own sections.

Steps to create integration record

Integration record

When creating a new Vulnerability Integration, several fields will need to be filled out.

Field

Description

Name

The display name of this import job. Typically includes the vendor name and the data type being imported.

Active

If checked, the job will run on its assigned schedule. If unchecked, the scheduler will not run the job, but users can still execute it manually at any time.

Run

Determines the type of schedule for the job (e.g. Daily, Weekly, On Demand). Depending on which type is chosen, additional fields will appear to choose the exact schedule.

Source integration

The vendor integration this job is a part of. This is a reference to the Third Party Integration record for the integration plugin.

Source instance

The integration instance this job belongs to. As there should be only one instance out-of-box, this should reference that instance. Inherent support for multi-source instances will be added in a future AVR release.

Integration script

A script include that contains logic for fetching the data. Discussed in detail in “Integration Request Script” section.

Integration factory script

Automatically generated script to instantiate the integration script. This should be left as its auto-generated value.

Report processor strategy

Either “Data Source Attachment” or “Custom Report Processor”. If “Custom Report Processor” is specified, a report processor script will need to be specified.

Report processor

A script include that contains logic for processing the data and creating/updating records in ServiceNow. Not required if the report processor strategy is set to “Data Source Attachment”.

Processor factory script

Automatically generated script to instantiate the report processor script. This should be left as its auto-generated value.

Is template

Sets if this record should be used as template in a multi-source environment. Multi-source support will be added in a future release.

Delay next process by (ms)

A delay (specified in milliseconds) after your job reports another page of data needs to be retrieved that the integration framework will wait before making the next Integration Process record. Can be used to limit the maximum request frequency of the integration job.

Note: Be sure to create Vulnerability Integration records from within the scope of your integration plugin. If another scope is selected, the “Application” field will be set to the wrong application, and the integration framework may not have the correct cross-scope access to execute your job.

Import Row set extension

Each type of record import (for example, Application Release, Scan Summary, or Application Vulnerable Item) will need its own table. This table should extend Import Row Set but does not otherwise need additional columns define.

Data sources

Data sources are used to parallelize imports. All Vulnerability Integrations will need at least one; adding more will increase parallelization, up to the number of available Background Schedule Workers available to the customer’s instance.

To do this, we use the ServiceNow platform capability for Data Source . Navigate to “System Import Sets” -> “Data Sources”. Click “New”. The type should be set to “File” and a format of XML or JSON, depending on your scanner API. It should be noted that the ServiceNow platform has better support for streaming XML, and therefore preferred over JSON when dealing with large numbers of records. It is also important that the table label end in “import” so it can easily be discerned from other tables we may add to the project. You will also need to provide an XPath, but since we will be parsing the attachment ourselves, set it to “/ignore/this/path” – this will cause the transform to skip exploding out the attachment to individual rows, and instead trigger a single row for the entire attachment.

Integration request script

Overview

The Integration Request script is responsible for constructing the request out to the scanner API, checking the HTTP status and results to determine if there was an error, and then saving a successful response as an attachment to the Integration Process.

This attachment will be processed asynchronously and is generally not handled in the same transaction as the download. See “Integration Processor Script” for more information.

Steps to create an integration request script

The Integration Request factory script is defined on the Integration’s “Integration script” field. This factory should pass the “integrationProcessGr”.

The script itself should extend “sn_vul.ApplicationVulnerabilityIntegrationBase”, which will handle the initialization of the script. The script will require a “retrieveData” function, which will act as the entry point.

The script will need to be able to handle incomplete responses or HTTP errors and mark the process as errored.

Lastly, a successful response should be saved as an attachment back to the Integration Process, using the command below:

new GlideSysAttachment().write(this.PROCESS, this.FILENAME, "xml", applicationContent)

Rest Message setup

It is advised to define any interactions with a scanner REST API as Outbound REST Message records, available in the navigator. These messages can then define needed parameters and formatting, and then use tokens for fields that may change based on configuration. The script include can then import the defined REST message and set just the needed parameters.

This will reduce the coding needed within scripts and allow for easier request troubleshooting and customization by customers and support.

Integration processor script

Overview

Data that has been downloaded from the scanner API as an attachment will be processed asynchronously within ServiceNow and will parallelized up to the number of data sources associated with the Integration.

The integration framework will associate the attachment with the next available data source and invoke the associated Transform Map.

While Transform Maps will frequently map the incoming fields directly, it is highly recommended to perform the parsing via script in order to stream the results, and then write the data using the AVR Import API, a script include that is part of Vulnerability Response.

The AVR Import API will invoke CI Lookup Rules for Application Releases, maintain the associations between Scanned Application, Application Release, Scan Summary, Application Vulnerable Item, Application Vulnerability Entry, CWEs and Package.

Once complete, the Integration Process is marked as Complete. Once all Processes in an Integration Run are complete, the Run is also marked as Complete.

Transform map

A Transform Map is a set of field mappings to translate the imported data into our target table. To create the Transform Map, open the “Create Transform Map” module under the “System Import Sets” menu.

The fields below should be set or reviewed:

Field

Description

Name

The Name of your Transform Map. It should include part of the integration name to make a clear distinction when viewing a list of Transform Maps.

Source Table

The table associated with your data source, which must extend Import Row Set.

Active

Flag indicating if the map will run. This must be set to True.

Run Business Rules

Flag indicating if the Business Rules on the target table will be run. Core Application Vulnerability functions depend on these Business Rules, and this must be set to True

Target Table

This should be set to the intended target table – for example, Application Release, Scan Summary, or Application Vulnerable Item. Note that parsing should be scripted, so this does not mean only this table will be written to.

Click “Submit” and you can start mapping columns. Navigate back to your Transform Map by navigating to “System Import Sets” -> “Transform Maps”. Open your newly created Transform.

While Transform Maps can write directly to the destination table via the “Field Maps” related list, it is highly recommended to script your parsing logic and use the AVR Import API script include to persist your data. This allows the platform to stream the data, which reduces memory consumption and increases average throughput. Using the API will also perform additional validation, set system fields automatically based on the calling integration, and lookup associated records and perform the association. To do this, only define an “onComplete” Transform Script.

This “onComplete” need only define the common Integration Framework method to invoke your integration’s defined processing script.

(function runTransformScript(source, map, log, target /*undefined onStart*/ ) { new sn_vul.VulnerabilityIntegrationHelper().processDataSource(import_set.getValue('data_source')); })(source, map, log, target);

Processor script include

The Processor Factory Script defined on the integration is invoked to retrieve the processing script include. This script include should extend “sn_vul.ApplicationVulnerabilityImportProcessorBase”, which will initialize the script include with the AVR Import API version, the associated Integration Process record, Source, and record counts.

The script must define the “process” function, which accepts an attachment object as a parameter. The script can then parse the attachment to create the API request parameter, which is then passed to the API.

The script will be responsible to check the response from the API for error conditions, and updating the run counts appropriately. Once done, the “completeProcess” function should be called.

Below is an example from the Veracode Application Vulnerability integration.

AVI API Overview

Application Vulnerability Response includes a set of scripts to provide common validation, reference lookup, and data persistence, referred to as the AVR Import API.

This should be retrieved via factory and must specify a version to use. As of this publishing, there is only v1, but as the interface changes, new versions will be added.

The public methods of this API include validation of minimum required fields, appropriate data types, and that commits were successful (for example, not aborted due to a business rule). Additionally, it will perform lookups for associated records – for example, when creating an Application Vulnerability Entry, only the CWE ID needs to be provided by the caller, not the internal sys_id, or creating placeholders if the CWE does not yet exist in the instance.

How to call API

By using a script that extends “ApplicationVulnerabilityImportProcessorBase”, the API instance does not to be constructed directly – it will already be set by initialize. However, if dealing with a custom solution of specific version of the API, the instance can be requested by providing a GlideRecord of the Integration Process to the AVR Import Factory, as shown below, and specifying a specific version:

var initParams = { 'process_gr': process }; this.AVR_API = new sn_vul.AVRImportAPIFactory().getAPI('v1', initParams); Once an instance is created, any of the public methods can be invoked by providing an object consisting of Name / Value pairs. In general, any non-system column for the target record can be specified, as well as additional “pseudo-fields”, which will be looked up and set during record update.

Please note that the API Constructor has new parameter (createClosed) for allowing creation of fixed AVITs on ingestion, apart from the system property value, that also controls the behaviour:

/** Control to create new closed AVITs from the scanner * createClosed property is expected from the integration with values: 'true' or 'false' to enforce behaviour * Falls back to the system property approach otherwise */l this.createClosed = initObj.createClosed ? initObj.createClosed : global.SecProperty.getProperty('sn_vul.create_closed');

The above field will be used to decide in the ingestion logic, if the new closed AVITs from the scanner would be ingested. The code snippet to control this would be:

if (avitGr.state == StateUtils.STATES.CLOSED && this.createClosed == 'false') { if (!avitGr.isValidRecord()) { result.inserted = false; result.unchanged = true; return result; } }

Fields that are not yet available in the target table may be specified, as well. These will generate warning messages but will not otherwise affect normal operation. If there is a column that is being added in a new release of Vulnerability Response, this mapping may be setup in advance.

For example:

var appObj = { source_app_id: attributes.app_id, app_name: attributes.app_name, source_additional_info: attributes.tags, source_assigned_teams: attributes.teams, description: 'Origin: ' + attributes.origin + ', Industry Vertical: ' + attributes.industry_vertical + ', App Type: ' + attributes.app_type }; var criticality = this.CRITICALITY[attributes.business_criticality]; if (criticality) appObj.business_criticality = criticality; var result = this.AVR_API.createOrUpdateApp(appObj);

Available methods:

createOrUpdateApp Insert or update an Application Release Can specify any column of “sn_vul_app_release” or “sn_vul_scanned_application” as part of the parameter’s attributes Requires: app_name source_app_id When passing in a single object, the API will use those fields for both the Scanned Application and Application release. However, if the fields between the two records should differ, a second parameter representing the Scanned Application can be passed. The first parameter will then only represent the Application Release. Ex: var result = this.AVR_API.createOrUpdateApp(relObj, appObj); createOrUpdateSummary Insert or update an Application Vulnerability Scan Summary Can specify any column of “sn_vul_app_vul_scan_summary” as part of the parameter’s attributes Creates stub for Application Release if not present Requires: source_app_id source_scan_id scan_summary_name last_scan_date createOrUpdateSummaryDetails Insert or update an Application Vulnerability Scan Summary Detail Can specify any column of “sn_vul_app_vul_scan_summary_details” as part of the parameter’s attributes. Requires: category_name severity scan_summary getAppReleases Returns a list of all Application Release records associated with the current integration instance

Error handling

The AVR Import API will return any errors that occur during data validation, normal operation, or rejected commits as an instance of an AppVulError object. This object will contain an error message, stack trace to root validation or error, and advice towards resolving the error.

During initial development, it can be useful to present any and all errors in order to troubleshoot parsing or data issues. Once approaching release, it is up to the integration to determine best how to handle the errors – either save them on the Integration Process record, handle them, or allow them to bubble up and mark the Integration Process as errored.

In this handling, though, it is recommended to avoid logging the underlying error and rethrowing, as this could potentially lead to the same error being logged multiple times.

Make sure to test with malformed or partial payloads, or payloads missing required fields, and verify that error handling logs or marks the error correctly, and that the process is not considered successful.

Best practices

Normal operation should not be generating any errors or warnings (unless an optional field has been added to the call in advance of an expected column change). When performing the import run, check the System Log to ensure the run is free of errors.

If the payload has a hierarchy, cache needed fields to avoid reparsing or multiple calls to the AVR Import API for the same record.

Additionally, as this can be a high-volume process, attempt to minimize informational logging.

Validation of processing

Processing scripts can be easily re-tested by opening an existing Integration Run, navigating to one of the child Integration Process records, then open the matched Data Source Import Queue record. Changing the status of this record back to “Queued” and saving will rerun the processing side of the integration job.

Verify that the target record and referenced records are created an associated correctly. The end result of the integration as whole (potentially over multiple integration jobs) is a Scanned Application, Application Release, Scan Summary, Application Vulnerable Item, and Application Vulnerability Entry.

The AVR API will invoke CI Lookup, State Mapping, and Normalized Severity. Verify that this lookup / mapping is being performed as expected.

CI Lookup Rules will control which Scanned Application are matched on an Application Release. This table is specific to Application Vulnerability Response, so it will likely not be populated for a new install. State Mapping will check the Source Remediation Status and set the state on the Application Vulnerable Item. Lack of mapping will be considered Open. Normalized Severity is set based on the Source Severity on the Application Vulnerability Entry. Verify both insert and update scenarios for the target records.

Check that the Integration Process and Integration Run are marked as Complete, and that the displayed counts are correct.

DevOps Integration

Third party integration record:

To support DevOps integration, in addition to the normal configuration the below fields should also be populated with appropriate values on the third-party integration (sn_sec_int_integration).

Pre-validation script Accepts a JSON of minimum configuration required to make an API call for credentials validation. Returns the response object of the API call which should include REST status code. Mainly used for the Security tool record creation process in DevOps. Validation script Accepts an implementation ID for which the credential validation needs to be done. Should make an API call with the config attached to the specified implementation ID and return the response object. Optionally can update the “valid” field of the integration record. Used for the hourly validation process of DevOps to make sure configuration is still valid. Sample from Veracode DevOps Integration:

Integration implementation:

In addition to the normal implementation, DevOps integration needs to handle the below scenarios.

DevOps integration implementation should retrieve and create/update the below data: Scan summary (sn_vul_app_vul_scan_summary) Scan summary details (sn_vul_app_vul_scan_summary_details) Category, Severity, Count, Scan Summary reference

Integration script should extend the base class DevOpsVulnerabilityIntegrationBase. Extending this will inherit the base methods required for the integration to run seamlessly. Implementation should handle the different combinations of input based on the Security tool. For example, in Veracode the various combinations can be: Application Name – Fetch the application ID and then get latest report. Application Id - Get latest report. Application Name , build version – Fetch the application ID, build ID, and then get report for this combination. Application Id , build Id - get report for is combination. Implementation should also handle the “Scan in-progress” scenario by validating the response of Scan summary API call and deferring the process to next integration run. Deferring can be achieved by passing an empty response without throwing any error in the integration script. Unlike all the other integrations, DevOps integration needs to run as close to real time as possible. Hence the “Run this script” part of the integration should periodically (1-5 minutes) do a real time check in the sn_vul_devops_integration_run table. Default implementation for this check has been added in VulnerabilityIntegrationUtils class.

Validation of DevOps Integration:

DevOps flow starts from an API exposed from Vulnerability Response application as mentioned below:

Script includes: VulnerabilityResponseDevOpsIntegration Method: getSummaryDetails.

This will populate the data into an intermediate table ‘ sn_vul_devops_integration_run ’ with state as ‘ new’ . Alternatively, data can be directly inserted into the same table.

Once data is populated, integration should pick up the data in next few minutes (depends on the scheduled time of integration), fetch the required data and associate the scan summary column with the created/updated record.

Any errors during this integration run should be propagated to the “ notes ” column in the same table. These notes can be used by DevOps Error handling to show an appropriate reason for failure.

Third party APIs:

These are some third-party APIs that can be used to fetch the required details for DevOps Integration for tools that are not integrated yet.

Note: These APIs are subjected to change based on the third-party tools’ discretion.

Checkmarx:

Checkmarx Cloud Base URL: US Env: https://api-sca.checkmarx.net EU Env: https://eu.api-sca.checkmarx.net Get Scans (for ScanId): /api/scans Get Scan status: /api/scans/{scanId} Get Scan summary for a scanId: /risk-management/scans/{scanId} Get Latest scan: api/projects/last-scan Checkmarx OnPrem Get Scans (for ScanId): Cxwebinterface/odata/v1/Scans Get Scan details for a scanId: Cxwebinterface/odata/v1/Scans({scanId}) Get results for a scanId: Cxwebinterface/odata/v1/results?$filter=ScanId+eq+{scanId}&$select=Id,ScanId,QueryId,Severity&$expand=Query($select=Name,CweId,Severity,CxDescriptionId;$expand=QueryCategories),Scan($select=ScanRequestedOn,IsIncremental,LOC)&$top=1 Get latest scan: Cxwebinterface/odata/v1/Projects({projectId})/LastScan Fortify on demand:

Get releases (filter by name to get release ID): /api/v3/releases Get release details: /api/v3/releases/{releaseId} Get scans: /api/v3/scans Get latest scan ID: /api/v3/scans?orderBy=completedOnEndDate&orderByDirection=DESC&limit=1 Get scan summary by ID: /api/v3/scans/{scanId}/summary