Software decomposition of container images using Kubernetes Visibility Agent and Syft - Support and Troubleshooting

Software decomposition of container images using Kubernetes Visibility Agent and Syft .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } }

Table of Contents

Scanning Container Images using Kubernetes Visibility Agent

Solution Architecture . 2

Pre-Requisites . 3

Store Applications: 3

Network Access . 3

Installation . 3

The Syft container image . 3

Helm Chart 3

K8s_informer.yaml 4

Configuration . 4

System Properties . 4

Credentials . 4

Secret of Type Docker Registry . 4

Accessing Private Repository in ECR (AWS Elastic Container Registry) 5

Accessing Private Repository in ACR (Azure Container Registry) 6

Accessing Private Repository in Google Artifact Registry . 7

Accessing Private Repository in Openshift . 8

Advanced Configuration . 9

The Informers Page and Form ... 10

The Informer Selection Algorithm ... 11

Fields Populated on cmdb_ci_docker_image . 11

Software Packages . 12

Platforms Supported .. 12

Scan Status . 12

Upgrades . 13

Scanning Container Images using Kubernetes Visibility Agent Kubernetes Visibility Agent is currently used to populate the CMDB with Kubernetes resources as well as minimal information about the container images used in the Kubernetes cluster

Customers need information on the container images such as:

Operating system and its version

Software and packages that are part of the image

Syft is an open-source tool used to scan container images. Syft is not detecting vulnerabilities and is looking only at the image itself. It does not have dependencies on any external databases and is totally independent.

ServiceNow already has integration with the Trivy CLI tool. This however requires a MID server and installation of the Trivy tool on that MID server. For customers that are not using Kubernetes Visibility Agent (KVA), the Trivy integration is the only option. For customers that do use KVA, this feature is the preferred option as it does not require any additional components and installations.

Syft was approved by legal in this 3 rd party ticket .

Solution Architecture When the user chooses to install KVA with the Syft integration option, the system will create an additional sidecar container within the informer pod.

This container will contain the Syft CLI tool that will be wrapped by code that accept scan instructions from the main informer and runs the scan. The results will be reported back to the main informer and from there via the ECC queue to the instance.

The Syft container exposes an http endpoint only to localhost, meaning only the informer container can call it. The scan instructions are sent using a synchronous http call from the main informer container to the Syft container.

On the instance, the system will populate the same model and status table as done by the Trivy flow.

The image scanning orchestration is managed by a script called Scan Container Images executed from a scheduled job that runs every minute and picks 10 images that were not scanned yet. If the scan of the previous batch has not completed yet, the system will not start a new batch. Based on the system properties described in later section, the system decides if scan is required, and if it should use KVA or MID. If KVA, the system will pick a KVA informer based on the informer selection algorithm described in the section below.

Pre-Requisites Store Applications: Kubernetes Visibility Agent version >=3.16.x

Discovery and Service Mapping Patterns version >=1.32.x (September 2026 release)

Informer Version >= 2.9.x

Network Access The informer pod should be able to access the image registries used by the customer. The Syft container will respect the proxy setting if such was configured (--set proxyUrl with the Helm installation)

Installation The Syft container image The Syft container image will be published in servicenowdocker/informer_syft:2.9.x

Customers that use their own private image registries should copy this image to their registries in the same way they copy the main informer's image.

Helm Chart When installing the informer with Helm chart, users should add the parameter:

--set runSyftScanner=true

K8s_informer.yaml We do not provide k8s_informer.yaml that supports the image scanning functionality.

Customers that do not want to use Helm for installations, should still use "helm template" command to render the Helm chart into a single Kubernetes manifest and then use this manifest for installation.

Configuration System Properties The following system properties should be set to true

sn_itom_pattern.container_image_scan

sn_acc_visibility.image_scan_enabled

The first one is telling the system to enable to image scanning mechanism. By default, this mechanism is using the Trivy CLI tool. The second one is telling the system to use the Syft scanner. Note that mixing the methods is not possible. Either use Trivy or Syft within Kubernetes Visibility Agent.

sn_acc_visibility.image_scan_exclude_artifacts - should we exclude packages from being reported to the instance based on regex. Default is true.

sn_acc_visibility.image_scan_exclude_artifacts_regex - Regular expression of packages that should be excluded and not reported to the instance. Default is:

apk|deb|rpm|portage|alpm

The regex is applied to the artifact.type field in the Syft SBOM.

sn_acc_visibility.scan_timeout_minutes - The number of minutes after which a request in status "in_progress" is declared as timed out and its status is changed to "error".

Credentials To access images in private repositories, we provide few options.

Secret of Type Docker Registry Create a secret in the same namespace as the informer with the credentials for accessing a specific registry. The system will read all named secrets of this type in the namespace and use the one that matches the image URL. The secret name should be provided in the Helm variable syftScanner.pullSecrets (see later in configuration options)

Example:

kubectl create secret docker-registry releases-repo-cred --docker-server registry-releases.devsnc.com --docker-username --docker-password --docker-email=tal.kapon@servicenow.com -n

Accessing Private Repository in ECR (AWS Elastic Container Registry)

EKS Clusters:

If the cluster is EKS, the preferred method is IRSA (IAM Roles for Service Accounts). With this method, the AWS administrator should grant access to registries to a given IAM role and tie this IAM role with a service account in a given namespace. Once this is done, the role is set on the informer's ServiceAccount using the helm parameter:

--set syftScanner.awsRegistryRoleArn=

Non EKS Clusters

Options are:

Use IRSA plus AWS pod identity webhook (Recommended) Use static credentials of the AWS User/IAM Role

IRSA with webhook (using community Helm chart, requires cert-manager:

# Install cert-manager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

# Wait for cert-manager to be fully ready before proceeding

# Now the Helm install

helm install amazon-eks-pod-identity-webhook \

jkroepke/amazon-eks-pod-identity-webhook \

--namespace kube-system \

--set config.defaultAwsRegion=

During the information installation:

--set awsSecretManagerRoleArn=

Static AWS Credentials:

If the Kubernetes cluster in which the informer is running is not EKS, the other way is to provide static credentials of the AWS User/IAM Role that is allowed to pull images from the registry.

The same user/IAM role should be allowed to pull images from all relevant accounts.

To provide the credentials, create a secret called aws-credentials in the same namespace as the informer. The secret should contain two keys: access_key and secret_key.

Example:

k ubectl create secret generic aws-credentials --from-literal=access_key= --from-literal=secret_key= --from-literal=role_arn= -n

Accessing Private Repository in ACR (Azure Container Registry)

AKS clusters:

When the informer runs in AKS cluster, the recommended way to use workload identity and grant access to relevant registries.

On the informer side, the serviceaccount and the pod should be annotated with client_id of the workload entity. This should be done by setting the helm parameter --set syftScanner.azureRegistryClientId=

For non-AKS clusters , customers have several options:

Define service principal and create a Kubernetes secret that holds its credentials Use workload identity on Azure side plus admission webhook on the informer side (recommended) Use static Azure user credentials

Define Service Principal:

If the Kubernetes cluster in which the informer is running is not AKS, customers should create service principal, grant AcrPull access on relevant repositories, and create a Kubernetes secret of type docker-registry. Example:

SP=$(az ad sp create-for-rbac --name $SP_NAME --skip-assignment)

SP_APP_ID=$(echo $SP | jq -r '.appId')

SP_PASSWORD=$(echo $SP | jq -r '.password')

# Assign AcrPull on the ACR

ACR_ID=$(az acr show --name $ACR_NAME --query id -o tsv)

az role assignment create \

--assignee $SP_APP_ID \

--role AcrPull \

--scope $ACR_ID

Then:

kubectl create secret docker-registry acr-secret \

--docker-server=talkaponacr1.azurecr.io \

--docker-username=$SP_APP_ID \

--docker-password=$SP_PASSWORD \

--docker-email=any@email.com \

-n

In case a single service principal is allowed to access all private repos in ACR, we should place in docker-server the string azurecr.io. The system will know how to prepend to that the requested registry name.

Workload Identity plus Webhook (for cluster with public K8s API server endpoint)

helm repo add azure-workload-identity \

https://azure.github.io/azure-workload-identity/charts

helm repo update

helm install workload-identity-webhook \

azure-workload-identity/workload-identity-webhook \

--namespace azure-workload-identity-system \

--create-namespace \

--set azureTenantID=" "

Note that the issuer configured on the federated entity should be the URL send on the Oauth request from the foreign cluster. In case of EKS, it looks like:

https://oidc.eks.us-east-1.amazonaws.com/id/F5FEB2039434BC0664E826BED1961FF6

Static Azure User Credentials

Create a Kubernetes secret of type docker-registry in the same namespace as the informer.

The docker-server would be azure.io

Docker-username would be the client_id

Docker-password would be the client_secret

Accessing Private Repository in Google Artifact Registry GKE clusters:

When the informer runs in GKE cluster, the recommended way to use workload identity and grant access to relevant repositories.

On the informer side, the serviceaccount should be annotated with google service account. This should be done by setting the helm parameter --set syftScanner. gcpRegistryServiceAccount=

For example:

--set syftScanner.gcpRegistryServiceAccount= gcr-pull-sa@deductive-reach-207607.iam.gserviceaccount.com

Following is an example on how to create a service account in GCP and establish trust with a Kubernetes service account.

// Creating the Google Service account

gcloud iam service-accounts create gcr-pull-sa \

--display-name="Artifact Registry Pull SA" \

--project=deductive-reach-207607

// Granting the google service account read access to the repository

gcloud artifacts repositories add-iam-policy-binding talkapon-gcp-repo \

--location=us-east1 \

--project=deductive-reach-207607 \

--member="serviceAccount:gcr-pull-sa@deductive-reach-207607.iam.gserviceaccount.com" \

--role="roles/artifactregistry.reader"

// Have the Google service account trust the Kubernetes service account servicenow-cnotal5 in namespace k8s-informer-cnotal5

gcloud iam service-accounts add-iam-policy-binding \

gcr-pull-sa@deductive-reach-207607.iam.gserviceaccount.com \

--role="roles/iam.workloadIdentityUser" \

--member="serviceAccount:deductive-reach-207607.svc.id.goog[k8s-infomer-cnotal5/servicenow-cnotal5]"

Accessing Private Repository in Openshift Openshift is coming out-of-box with a secret called pull-secret in the namespace openshift-config. This secrets provides access to well-known Redhat registries like registry.redhat.io

When the informer runs in Openshift, use the flag --set syftScanner.runningInOpenshift=true

This will grant the informer access to the secret pull-secret in the namespace openshift-config.

When the informer runs in non Openshift cluster, copy the secret to the informer's namespace.

Advanced Configuration Helm parameter

Description

Default value

syftScanner.memoryRequest

The memory requested for the syft scanner container

100Mi

syftScanner.memoryLimit

Memory limit for the syft scanner container

1Gi

syftScanner.imageMaxSizeBytes

Maximum size (in bytes) of an image we are willing to scan. Images larger than this are skipped.

Note: the total size of an image may exceed the container memory limit since layers are downloaded and scanned one by one.

2147483648

syftScanner.cpuRequest

CPU request for the syft scanner container

100m

syftScanner.cpuLimit

CPU limit for the syft scanner container

300m

syftScanner.queueMaxSize

Maximum number of pending image scan requests

1000

syftScanner.timeoutSec

Maximum time in seconds to wait for Syft to scan an image

600

syftScanner.maxRetry

Maximum number of times the informer retries a scan request after the syft sidecar rejects it with a 429 (Retry-After). Once this many retries are exhausted, the scan request is aborted and reported as a failure.

5

syftScanner.enrich

Controls syft's --enrich flag, which enables online enrichment of catalog results by fetching extra metadata from upstream sources. "all" turns on every enricher; it can also be a comma-separated list of specific enrichers (e.g. "java,golang") or "none" to disable enrichment entirely.

all

syftScanner.parallelism

number of cataloger workers to run in parallel

1

syftScanner.gogc

Sets the GOGC garbage-collection target for the syft Go process. Lower values make the GC run more frequently, reducing peak memory at the cost of CPU. Left empty by default so Go's built-in default (100) applies; set an explicit value (e.g. "20") only when tuning memory usage.

None

syftScanner.searchRemoteLicenses

When true, syft resolves Go module licenses by fetching/cloning their source repositories over the network. Disabled by default because cloning large upstream repos (e.g. oracle/oci-go-sdk) into memory causes dramatic memory spikes; turning it off only drops the license field on Go packages, not their presence, version, or PURL. (Overrides what enrich=all would otherwise enable.)

false

syftScanner.azureRegistryClientId

The client ID of the Azure user-assigned managed identity that has AcrPull permissions on the target ACR. Setting this turns on Azure Workload Identityfor the syft-scanner container.

None

syftScanner.awsRegistryRoleArn

ARN of the AWS IAM role with permission to pull from the target ECR registries. The role must trust the EKS pod service account. Setting this adds an eks.amazonaws.com/role-arn annotation to the informer ServiceAccount.

None

syftScanner.gcpRegistryServiceAccount

Email of the GCP IAM service account that has Artifact Registry Reader permissions. The GKE pod service account must be bound to this GCP SA via Workload Identity. Setting this adds an iam.gke.io/gcp-service-account annotation to the informer ServiceAccount.

None

syftScanner.pullSecrets

Comma-separated list of Secret names (in the informer's own namespace) that the informer is allowed to read as docker-registry pull credentials. The informer Role is scoped to `get` on exactly these names (via resourceNames), so only these secrets can be read. Each is fetched by name and used only if it contains a .dockerconfigjson key.

IMPORTANT: When providing value in the Helm command every comma should be escaped with a backslash. For example:

helm install k8s-informer k8s-informer-chart --set syftScanner.pullSecrets="secret1\,secret2\,secret3"

aws-credentials,azure-credentials,gcp-credentials,pull-secret

syftScanner.openShiftPullSecret.enabled

When enabled, the informer reads the OpenShift pull-secret and adds its credentials to the registry auth chain used by the syft scanner. This enables pulling images from registry.redhat.io, quay.io, and other Red Hat registries on OpenShift clusters. The informer first looks for a secret specified in the parameter `secretName` in its own namespace; if that lookup fails it falls back to the cluster-wide openshift-config/pull-secret (the chart also renders a Role + RoleBinding granting get on that single named secret in openshift-config).

false

syftScanner.openShiftPullSecret.secretName

Name of the dockerconfigjson secret in the informer's own namespace to use as the primary source for the OpenShift pull-secret.

None

syftScanner.openShiftPullSecret.runningInOpenshift

Set to true only when installing on an OpenShift cluster. When true, the chart also renders a Role + RoleBinding granting `get` on openshift-config/pull-secret. On non-OpenShift clusters this must remain false, otherwise the install fails because the openshift-config namespace does not exist

None

The Informers Page and Form We are adding two new fields to the informer record. Both will be hidden by default in the informers list and shown in the Informer's form.

Image Scanner Enabled - values are true/false. This field is showing which informers are configured to run image scan.

Image Scan Registries - this is comma separated list of image registries the informer declared are covered via credential-less trust relationship. Optional values are: ecr, acr, google. If the field is not empty, and the instance needs to scan an image in one of those registries, the system will send the instruction to one of the informers that declared on it. If no informer declared on it, the instruction will be sent to an arbitrary informer that supports image scanning. See more details in the section "The informer selection algorithm".

When an informer is configured to scan images a new UI action called "Get Syft Scanner Logs" will be available. When pressing on this action, the system will request the informer to send the logs. The logs will be attached to the form as file named k8s_informer_syft.log.

The Informer Selection Algorithm The informer will report it supports a certain cloud registry via credential-less trust relationship based on the following parameters:

If syftScanner.awsRegistryRoleArn is set: ecr

If syftScanner.azureRegistryClientId is set: acr

If syftScanner.gcpRegistryServiceAccount is set: google

If more than one of the above parameters is set, then the value reported will be comma separated.

The backend will select an informer for scanning a given image based on the following decision flow:

Fields Populated on cmdb_ci_docker_image The following fields are populated by the software on cmdb_ci_docker_image. Note that not all fields might be available on every image.

os (Label "Operating System").

os_version (Label "OS Version")

architecture (Label "Architecture"). Examples: amd64, arm64

size_bytes (Label "Size (bytes)")

os_family (Label "OS Family"). Examples: rhel, centos, alpine, azurelinux, windows

image_created_at (Label "Image created")

command (Label "Command")

Software Packages As with the Trivy integration, software packages contained in images are modeled in table sn_itom_pattern_container_image_os_packages and show in the related list of cmdb_ci_docker_image:

Important Note: We rely on Syft to provide the information on maintainer and version. We cannot guarantee that those fields will be populated in all cases.

Platforms Supported The scanner will support both Linux based images and Windows based. However, the support for Windows images is partial because of Syft limitations. We do not support Windows images that contain non-distributable base layers. With other images, Syft catalogers have partial support for Windows.

Scan Status The table sn_itom_pattern_container_image_scan_status continues to maintain the scan status of images, as with the Trivy integration.

Upgrades Helm upgrades can be executed using the helm upgrade command as before.

Upgrade from the instance is possible only if the previous version was installed using helm and already installed using the runSyftScanner option.

A new property is added to the Kubernetes Visibility Agent properties screen. Users should fill in Syft container box, if the location of the desired image is not in the same location as the main informer image, while adding the "_syft" suffix to the repository name.

For example:

If the main informer image location is "my_org/informer:2.9.0" and the syft image is "my_org/informer_syft:2.9.0", there is no need to fill in this field. Only if the location is different, then this field should be edited.