CNO for Visibility (Informer) extensibility and customization - Support and Troubleshooting

CNO for Visibility (Informer) extensibility and customization . 8

Table of Contents Goals Pulling Additional Resources Default Information Pulled by the System Filtering Additional Resources Custom Information Input Config Files Custom Relations Custom Related Entries Processing the Information on the Instance Filtering Out Resources on the Instance Scripted API for Retrieving Resource Information On-Demand Goals The goal of this feature is to enable customers to bring into their database additional resources and attributes from the Kubernetes environment and process them according to their needs. In addition, it provides capabilities for manipulating and filtering the resources reported by the informer. This document first explains the information the system can pull, followed by concrete examples of the configuration files.

Pulling Additional Resources Users may declare on any Kubernetes resource to pull into the CMDB. The resource definition will include the APIVERSION and NAME as seen in the output of the command kubectl api-resources .

Default Information Pulled by the System By default, the system will pull the following information:

Name Namespace (for namespaced resources) Kubernetes UID API Version Kind Labels and Annotations By default, the system will store the information in the cmdb_ci_kubernetes_component class.

The system will create a relation and a reference from the cmdb_ci_kubernetes_component to the cluster CI.

In case the resource is namespaced, the system will create a relation to the namespace CI.

If you would like to place the resource in a dedicated new table, it will be possible to declare which is the target table. The target table should extend cmdb_ci_kubernetes_component and should have an identification rule that inherits the one from the parent class.

Filtering Resources Users may declare on a subset of the resources of a given kind to bring to the CMDB. For example, you may request to bring ConfigMap resources whose name is "version".

The filtering is done by specifying a JSON path of the field in the resource JSON whose value should be inspected, and a regular expression applied to this value. It is possible to define multiple such conditions. If any of the conditions show a match, the resource will be sent to the CMDB.

The filtering mechanism is applicable both to out-of-the-box resources or to an additional resources defined by customers.

Custom Information Users may define additional information to retrieve from a given resource whether this is a resource CNO brings out-of-the-box or an additional resource.

Per each resource, it is possible to define any number of custom fields. Per each such field, you should define the field name and JSON path to the information within the resource JSON and optional regular expression to extract part of that value.

By default, the system will store the information in the "attributes" field of cmdb_ci_kubernetes_component as a key/value in a JSON string. You can define a custom target field name in the relevant table in the CMDB.

If the aggregated size of the ‘attributes’ field exceeds 65K bytes (which is the field size in the database), the data will not be sent to the instance.

Input Config Files The helm chart will now include an additional value with two sub-keys:

additionalResources:

resources:

mappings:

Both sub-key values will be loaded from YAML files with the following structures:

The "resources" part is a YAML array, where each element contains arrays of apiGroups, apiVersions and resources. Example:

- apiGroups :

- ''

apiVersions :

- v1

resources :

- configmaps

- apiGroups :

- config.openshift.io

apiVersions :

- v1

resources :

- clusterversions

In this example, we request the system to bring into the CMDB v1/configmaps and config.openshift.io/v1/clusterversions

When using the ClusterRole provided in the Helm chart, the system will pull the resource list into the role definition and add the verbs list, get and watch.

With the example above, the system will create a cmdb_ci_kubernetes_component CI per each ConfigMap and ClusterVersion resource in the cluster.

The "mapping" part is a YAML map, where each key represents a resource, and sub-keys represent fields ,optional target class and filters. Example:

v1/configmaps :

targetClass : "u_cmdb_kubernetes_configmap"

filter :

- jsonPath : ".metadata.name"

regex : ^version$

fields :

- fieldExtractor :

jsonPath : ".data.clusterFqdn"

name : cluster_fqdn

targetColumn : "u_cluster_fqdn"

config.openshift.io/v1/clusterversions :

fields :

- fieldExtractor :

jsonPath : ".status.desired.version"

name : version

In this example:

The system will bring into a custom table u_cmdb_ci_kubernetes_configmap all ConfigMap resources in the cluster whose name is "version". It will extract from the ConfigMap the key "clusterFqdn" and place the value in a field called u_cluster_fqdn

The system will also bring resources of kind ClusterVersion and place them in the cmdb_ci_kubernetes_component table. It will extract the value status.desired.version and place it in a JSON map in the ‘attributes’ column under a key called “version”.

Both parts are optional, but each resource listed in the "mappings" part, should be also listed in the "resources" part . Within the mappings part all parts are optional.

Within "fields", targetColumn is optional

Within each "fieldExtractor", "regex" is optional

Within "filter", "regex" is optional

The excludeFromResource Key By default, the system loads the full information brought from Kubernetes API server into the informer's memory. In case of large resources, we provide the option to reduce the memory footprint by specifying JSON paths of parts of the resource that should be excluded from the saved object.

In the mappings definition, under a specific object definition add the key excludeFromResource and underneath place an array of excluded JSON paths.

Example:

excludeFromResource :

- ".report.components"

components"

To set resources.yaml while running helm command, use the following flag:

--set-file additionalResources.resources=/path/to/resources.yaml

To set mappings.yaml while running helm command, use the following flag:

--set-file additionalResources.mappings=/path/to/mappings.yaml

Note: If you are using k8s_informer.yaml, resources and mappings values have to be provided as a json string. An example has been provided in k8s_informer.yaml for reference.

Custom Relations On top of the default relations created by the system, you can define additional relations between the additional resources and other Cis in the Kubernetes cluster. The system requires the user to define:

JSON path and optional regex to the target resource kind JSON path and optional regex to the target resource UID or JSON path and optional regex to the target resource namespace and name The relation type (e.g. Contains::Contained by) The relation direction (optional) – true/false with default is true. True means the parent of the relation would be the additional resource and the child the CI identified by the declared JSON paths.

Examples of definitions in the mappings.yaml

Lookup by UID

We would like to create a relation from resource controller revisions to its owners. In this case the system will look up for the owner using its UID

apps/v1/controllerrevisions :

relations :

- targetResourceUid :

jsonPath : ".metadata.ownerReferences[:].uid"

targetKind :

jsonPath : ".metadata.ownerReferences[:].kind"

relationType : Controller for::Controlled by

relationDirection : true

Lookup by Namespace and Name

We would like to create a relation from ClusterRoleBinding to the corresponding ServiceAccount. ServiceAccount is namespaced, so we provide JSON path to the name and namespace

In addition, we would like to create a relation to the ClusterRole. ClusterRole is not namespaced so we provide only the JSON path of the name

rbac.authorization.k8s.io/v1/clusterrolebindings :

relations :

- targetKind :

jsonPath : ".subjects[:].kind"

targetResourceName :

jsonPath : ".subjects[:].name"

targetResourceNamespace :

jsonPath : ".subjects[:].namespace"

relationType : Used by::Uses

relationDirection : false

- targetKind :

jsonPath : ".roleRef.kind"

targetResourceName :

jsonPath : ".roleRef.name"

relationType : Used by::Uses

relationDirection : false

Important note: At this point, the system will not delete relations which were created in the past but are not discovered any more due to changes in the declaration or in the object data.

Custom Related Entries Related entries are records in a non-CMDB table which has a reference to a CMDB record, in this case a CI that extends cmdb_ci_kubernetes_component. Users can define a custom table which will hold those records and populate it using data coming from the Kubernetes API server.

The definition will be in the mapping.yaml and includes:

targetTable: the name of the table of the related records referenceField: the name of the field holding the reference to the patent cmdb_ci_kubernetes_component record arrayJsonPath: JSON path of an array object within the payload returned by the Kubernetes API server when retrieving the information of the parent Kubernetes resource. Fields: array of fields to be populated in the related entry. Each field contains: name: a friendly name targetColumn: the column in the target table fieldExtractor: jsonPath: how to extract the field from each element in the array regExp (optional): extract the value from the original field using regular expression filter (optional): map of field names to regular expression. If this is provided, the system will bring only related entries whose value is full matched by the regular expression Note that the custom related table should be added as a related entry to the parent class identification rules.

Example:

rbac.authorization.k8s.io/v1/clusterroles :

targetClass : cmdb_ci_kubernetes_cluster_role

relatedEntries :

- targetTable : u_cluster_role_rule

referenceField : u_cluster_role

arrayJsonPath : ".rules.*"

fields :

- name : api_groups

targetColumn : u_api_groups

fieldExtractor :

jsonPath : ".apiGroups"

- name : resources

targetColumn : u_resources

fieldExtractor :

jsonPath : ".resources"

- name : verbs

targetColumn : u_verbs

fieldExtractor :

jsonPath : ".verbs"

Filter condition should be added if there is a need to bring only part of the related entries. In this example, add the filter below in order to bring only rules that include write verbs.

rbac.authorization.k8s.io/v1/clusterroles :

targetClass : cmdb_ci_kubernetes_cluster_role

relatedEntries :

- targetTable : u_cluster_role_rule

referenceField : u_cluster_role

arrayJsonPath : ".rules.*"

filter :

verbs : '.*(create|delete|update|patch).*'

fields :

- name : api_groups

targetColumn : u_api_groups

fieldExtractor :

jsonPath : ".apiGroups"

- name : resources

targetColumn : u_resources

fieldExtractor :

jsonPath : ".resources"

- name : verbs

targetColumn : u_verbs

fieldExtractor :

jsonPath : ".verbs"

Processing the Information on the Instance In addition to the traditional business rules, ServiceNow provides hooks for inspection, manipulation, and post-processing of the IRE (Identification and Reconciliation Engine). The capability will be similar to the pattern pre/post script mechanism:

Once the informer sends payloads to the instance, it is possible to handle it in several ways:

Use a traditional business rule to hook into a specific change and act upon it. Note however, then when a CI is inserted, relations are still not there, and references are not yet populated. A pre-sensor script accepts input the IRE payload as a string and can examine and manipulate it. The script returns either the original payload or a manipulated one. This option should be used with care since the script will work on every payload and as such should be very efficient and avoid unnecessary database calls. To use this option navigate to "Pre Post Processing", press "New" and select in the Patterns field the pattern named "K8s Informer Place Holder Pattern for Pre Post Scripts". Then select "Pre sensor" in "When to execute" and write your script in the "Script" field. A post-sensor script takes as an input the IRE output string and can take additional actions. Navigate to "Pre Post Processing", press "New" and select in the Patterns field the pattern named "K8s Informer Place Holder Pattern for Pre Post Scripts". Then select "Post sensor" in "When to execute" and write your script in the "Script" field. Below is an example of a pre-sensor script. It picks the version information from the ClusterVersion CI and places it in the comments field of the cluster CI.

var rtrn = {};

// Parsing json string to object

var payloadObj = JSON . parse ( payload );

var cluster = payloadObj . items [ 0 ];

// Look for the ClusterVersion item

var found = false ;

for ( var i = 0 ; i payloadObj . items . length ; i ++ ) {

if ( payloadObj . items [ i ]. className != 'cmdb_ci_kubernetes_component' )

continue ;

var kind = payloadObj . items [ i ]. values . kind ;

var attributes = payloadObj . items [ i ]. values . attributes ;

if ( 'ClusterVersion' == kind && attributes ) {

var attrObj = JSON . parse ( attributes );

if ( attrObj && attrObj . version ) {

cluster . values [ 'comments' ] = attrObj . version ;

found = true ;

}

break ;

}

// Clearing the payload string to save memory

var message = 'Version not found' ;

if ( found ) {

payload = JSON . stringify ( payloadObj );

message = 'Version found' ;

}

rtrn = {

'status' : {

'message' : message ,

'isSuccess' : true

},

'patternId' : patternId ,

'payload' : payload

};

Filtering Out Resources on the Instance A pre sensor script can be used to eliminate unwanted resources from the payload sent by the informer. The script below takes an IRE (Identification and Reconciliation engine) and removes items of a given class along with the relevant relations and references. The function removeClass takes a payload object and a class name and returns the modified payload.

var K8sInformerIREHelper = Class . create ();

K8sInformerIREHelper . prototype = {

initialize : function () {

},

/**

* Input: IRE payload object

* class name of items that should be removed

* Output: IRE object

*/

removeClass : function ( payloadObj , sysClassName ) {

var mapOldToNew = {};

var removedItemsCount = 0 ;

var clusterName = '' ;

if ( payloadObj . items . length != 0 )

clusterName = payloadObj . items [ 0 ]. name ;

// Iterate the items and move to the result only items that do not have the defined class

var newPayload = { items: [] , relations: [] , referenceItems: [] };

for ( var itemCount in payloadObj . items ) {

var item = payloadObj . items [ itemCount ];

if ( item . className != sysClassName ) {

newPayload . items . push ( item );

mapOldToNew [ itemCount + '' ] = itemCount - removedItemsCount ;

} else {

removedItemsCount ++ ;

}

// Iterate the relations and move to the result only relations whose both parent and child were moved to the

for ( var relCount in payloadObj . relations ) {

var relation = payloadObj . relations [ relCount ];

var newParent = mapOldToNew [ relation . parent + '' ];

var newChild = mapOldToNew [ relation . child + '' ];

if ( this . isOK ( newParent ) && this . isOK ( newChild )) {

var newRelation = { parent: newParent , child: newChild , type: relation . type };

newPayload . relations . push ( newRelation );

}

// Iterate the references and move to the result references whose both referenced and referencedBy were moved to the

for ( var refCount in payloadObj . referenceItems ) {

var ref = payloadObj . referenceItems [ refCount ];

var newReferenced = mapOldToNew [ ref . referenced + '' ];

var newReferencedBy = mapOldToNew [ ref . referencedBy + '' ];

if ( this . isOK ( newReferenced ) && this . isOK ( newReferencedBy )) {

var newRef = { referenced: newReferenced , referencedBy:newReferencedBy , referenceField: ref . referenceField };

newPayload . referenceItems . push ( newRef );

}

gs . info ( 'K8sInformerIREHelper: ' + removedItemsCount + ' items removed from IRE payload. Cluster: ' + clusterName );

return newPayload ;

},

isOK : function ( index ) {

var indexStr = index + '' ;

return global . JSUtil . notNil ( indexStr ) && indexStr != 'undefined' ;

},

type: 'K8sInformerIREHelper'

};

This script can be called from a pre-sensor script. In the example below, we call the script to remove all occurrences of cmdb_ci_docker_container from the payload.

var rtrn = {};

// parsing the json string to a json object

var payloadObj = JSON . parse ( payload );

// Clearing payload string to save memory

payload = null ;

payloadObj = new K8sInformerIREHelper (). removeClass ( payloadObj , 'cmdb_ci_docker_container' );

rtrn = {

'status' : {

'message' : 'Informer IRE was processed' ,

'isSuccess' : true

},

'patternId' : patternId ,

'payload' : JSON . stringify ( payloadObj )

};

Scripted API for Retrieving Resource Information On-Demand When CNO is deployed, a scripted API can be used to send a request to the informer to get the information on a specific Kubernetes resource. See KB1648272