CNO - MID frequent restarts - Support and Troubleshooting

CNO - MID frequent restarts The background MID container is created with memory resources defined in the table sn_cno_resources .

If Metric Intelligence plugin is not installed, the CNO uses by default the small memory model, which sets the memory value to 1,200 MB.

MID JVM memory max size should not exceed the container memory size and by default it is set as following: container a memory size multiplied by the ratio 0.8. So currently it is 960 MB.

We found out, that the above memory settings doesn’t meet, if customer's cluster is busy. When the cluster discovery is running the MID memory is growing beyond the allocated limit of the container memory, this causes the Kubernetes to kill the container.

The procedure below suggests how to change the resources settings.

How to fix existing deployments For the existing deployments the changes may be done on the cluster itself.

We suggest doing the changes first on one cluster and then if this is successful, repeat the same on the rest of the clusters.

Change MID container memory limit on a cluster with the command: kubectl edit statefulset sn-acc-mid -n servicenow

This will open vi editor. Navigate to definitions of container memory resources and change it to 2Gi as it is shown on the snapshot below:

Run the cluster discovery: Navigate to the Discovery Schedules Set filter *Kubernetes. Add MID Server column to the view.

Open the schedule entry for the MID of the cluster that was edited in the previous step and press ‘Discover now’.

After pressing on ‘Discover now’ the discovery status will be popped up. Open its record and wait for discovery finish. If the step 2 is successful, repeat steps 1,2 for the rest of the clusters.

Note: Alternative to edit the statefulset, the CNO may be redeployed after making the necessary changes on the instance itself, like this is explained below.

The instance changes The changes on instance should be done for new deployments or redeploying of existing deployments.

Navigate to the table sn_cno_resources and edit the record of SN_ACC_MID container for Small memory model as following: Memory Limit: 2 Memory Limit Unit: Gi

Navigate to the sys_properties table and create the system property sn_k8s.mid_wrapper_memory_allocation_ratio with the value 0.6: Select the application: Cloud Native Operations Set the suffix: mid_wrapper_memory_allocation_ratio Set the type: string Set the value: 0.6