Running Kubernetes Visibility Agent (CNO for Visibility) at scaleRunning Kubernetes Visibility Agent (CNO for Visibility) at Scale When you run Kubernetes Visibility Agent (a.k.a CNO for Visibility) on large number of clusters (>100) we recommend to tune your configuration so that no data coming from the informers will not be lost. Setting the Maximum Number of Concurrent Worker Threads By default, the system will allow up to 4 worker threads to process payloads coming from the informers simultaneously. When having large number of informers, we recommend to change this number to 8. Edit the system property sn_acc_visibility.k8s_informer_max_worker_threads and change the value to 8. Turning off The Continuous Discovery If a near real-time CMDB is not mandatory on some or all of your clusters, it is recommended to turn off the continuous discovery on those clusters. This will reduce the churn in your CMDB. Turning off the continuous discovery can be done in several ways: During the installation of the informer: When using Helm: use --set continuousDiscovery=falseWhen using k8s_informer.yaml: change the value of the environment variable CONTINUOUS_DISCOVERY to false From the instance: Configure all informers: Navigate to CNO for Visibility/Configuration Parameters, add a new record and select 'Continuous Discovery' in the parameter name Configure a specific informer: Navigate to CNO for Visibility/Home, click on the relevant informer. In the "Kubernetes Informer Parameters" section, click new to add a new record and select the 'Continuous Discovery' in the parameter name. Do not Ingest Container Configuration Items (from Informer version 2.4.x) For most use cases, the container configuration items have no value. The container images are the important CIs which can tell you what images are being used and when. On the other hand, containers come in large volume and are created and deleted often. You can turn of the ingestion of containers (from version 2.4.x of the informer) by setting the flag --set createContainers=none when using Helm chart, or changing the environment variable CREATE_CONTAINERS in the k8s_informer.yaml. Changing the Full Discovery Interval In case of informers running with continuous discovery on, it is recommended to reduce the full discovery interval to once every few days. Note that time interval is specified in minutes. This can be done in several ways: During the installation of the informer: When using Helm: use --set fullDiscoveryMin=5760When using k8s_informer.yaml: change the value of the environment variable FULL_DISCOVERY_MIN to 5760 From the instance: Configure all informers: Navigate to CNO for Visibility/Configuration Parameters, add a new record and select 'Full Discovery Duration' in the parameter name Configure a specific informer: Navigate to CNO for Visibility/Home, click on the relevant informer. In the "Kubernetes Informer Parameters" section, click new to add a new record and select the 'Full Discovery Duration' in the parameter name. Limiting the Number of Concurrent Full Discovery Runs By default, the system limits the number of concurrent full discovery runs to 5. Consider reducing the number to 2 or 3 in order to limit the possible spikes in data processing on the instance. Edit the system property sn_acc_visibility.max_concurrent_full_discovery to reduce the number. Using the Latest Informer and Backend Versions The recent versions (Backend 3.9.x, 3.10.x and Informer 2.3.x, 2.4.x) introduced efficiency and scale improvements. Specifically, we added the capability to reduce reporting of flapping and short-lived changes in resources. Increasing the Allowed Delay for Processing Payloads By default, the system will discard payloads waiting for processing for more than one hour. In extreme cases, we might accumulate a backlog which is waiting for processing more than that. To increase the allowed delay edit the system property sn_acc_visibility.informer_ecc_max_delay_sec. Change the value to 7200.