Suppressing frequent changes in Kubernetes clusters reported by Kubernetes Visibility Agent (formerly CNO for visibility)What is the Suppression of Frequent Changes? Kubernetes Visibility Agent (formerly CNO for Visibility) is reporting in near real-time to the ServiceNow instance on changes that happen in the Kubernetes cluster. However, certain changes in the clusters are happening repeatedly and frequently. Reporting on those changes can create a load on the instance, and does not add any value. A few examples of such changes: A CronJob configured to run every minute will create a new Job and Pod every minute and delete them later onA pod that fails to start because of some problem (e.g. fails to pull an image) will see a frequent changes in its statusWhen a container is crashing and restarted because of a frequent problem "Kubernetes Visibility Agent" can ignore changes on Kubernetes resources that happen in a rate the exceeds a configured threshold. Turning Off the Feature The feature is enabled by default but can be turned off by users: When installing using Helm chart, use the command line argument --set suppressFrequentChanges=false to disable the feature. When using the Kubernetes Yaml file, change the value of the environment variable SUPPRESS_FREQUENT_CHANGES to false The Change Rate Threshold When the suppression feature is turned on, CNO for visibility will suppress changes that happen in a rate of more than 0.15 time minute (roughly more frequent than every 7 minutes). This threshold can be controlled by users. When installing using Helm chart, use the command line argument --set suppressChangesRatePerMin=<Number in float format> to set the desired threshold. When using the Kubernetes Yaml file, change the value of the environment variable SUPPRESS_CHANGES_RATE_PER_MIN to the desired value in float format. What are Short Lived Resources? Short lived resources are resources that live in the Kubernetes cluster for a short period of time. Reporting on each of such resource can create a load on the instance and has little value. Starting from informer version 2.4.x, "Kubernetes Visibility Agent" will report by default on a given percentage of additions/modifications of resources which whose predicated life span is short. The system predicts the life span of a given resource based on statistics collected on similar resources in the last day (since the last full discovery cycle). The feature is applicable to pod and jobs. Pods are considered similar for that purpose, if they share the same container images, are in the same namespace, and have the same name prefix (the part before the first dash) Job are considered similar if their spec.template.spec is identical, they are in the same namespace, and have the same name prefix. The feature is controlled by two configuration parameters: Helm Value Env Variable Default suppressChanges.resource_life_span_minutes SUPPRESS_CHANGES_LIFE_SPAN_MINUTES 15 We consider a resource to be short-lived if the expected life span is below this number in minutes. 0 means the feature is disabled suppressChanges.suppressRate SHORT_LIVED_SUPPRESS_RATIO 10 When a resource is short-lived, we will report on one of N additions, where N is this number Both parameters can be overridden from the instance.