Understanding Kubernetes Visibility Agent ECC Queue records, and Sensor business rules - Support and Troubleshooting

Understanding Kubernetes Visibility Agent ECC Queue records, and Sensor business rules .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Table of Contents Introduction ECC Queue record fields Sensor Business Rule, and Events Introduction This article aims to help with triaging performance issues of MID Servers and ECC Queue, and debugging the steps of the feature when Kubernetes Visibility Agent sends data to the instance.

ECC Queue record fields Column Queue

Value

Topic input/output

k8s_informer

Agent input/output

k8s_informer_

Source input/output

k8s_informer_

Name output k8s_informer_ , where command name is:

restart_informer pause_informer resume_informer full_discovery grab_logs input k8s_informer Payload output Command for the Informer. e.g.

restart_informer reload_parameter:pause=true reload_parameter:pause=false full_discovery grab_logs input 'MESSAGE_SEPARATOR' separates the messages within the paylaod, and within each message 'TYPE2PAYLOAD_SEPARATOR' splits it into its 2 parts. The first of those parts is the Message Type. e.g. Heartbeat in this example: MESSAGE_SEPARATORHEARTBEATTYPE2PAYLOAD_SEPARATOR{"version":"k8s_informer_simulator","repository":"informer_simulator_repo","continuous_discovery":true,"pod_name":"informer_simulator_pod","upgrade_status":""}

Message Types:

HEARTBEAT INFORMER_DETAILS FULL_DISCOVERY_STARTED GRAB_LOGS DISCOVERY_STATUS 4faed24ad5660110f8779638418f41c8 / e368e7534748259029325d22736d43f8 = Discovery, where there are 2 IRE payloads separated by '@@@@@@@@@@', where the first is the IRE payload to insert/update and the second are the CI items to delete. State + Processed input The State is set to Processed, and the Processed timestamp populated, but only when the sysevent has finished processing. e.g. after a full_discovery input has been through IRE and updated the CMDB.

Error String input 'record too old' will be seen if by the time the event is processed, the message is more than 1 hour old (sn_acc_visibility.informer_ecc_max_delay_sec).

Sensor Business Rule, and Events 'Handle standalone k8s_informer message' business rule runs for all topic=k8s_informer ecc_queue inputs. It runs after insert, and just fires a sysevent with a name sn_acc_visibility.k8s_informer.message_ .

By offloading this to an event, the sensor code is not actually processed by the ecc_queue insert API_INT semaphore thread at all. This way allows more control over the appnodes assigned to processing these events, and numbers of events processed concurrently.

There is a 'Handle the standalone k8s_informer event' Script Action for each event name/queue number. These call script include K8sInformeEccInputHandler function handleEvent.

There are separate Event Queues, named sn_acc_visibility.k8s_informer.message_ , where each event queue has its own sys_trigger job named Handle k8s_informer_events_ , e.g. k8s_informer_events_10

System property sn_acc_visibility.k8s_informer_max_worker_threads defines the maximum number of concurrent worker threads that will process the k8s informer payloads.

For further information, see: KB1580241 Controlling the number of "Kubernetes Visibility Agent" (formerly CNO for Visibility) concurrent worker threads

That covers:

Controlling the Maximum Number of Concurrent Worker Threads Routing ECC Messages to Queues Custom Routing of ECC Messages