CMDB CI Deletion Jobs for High Volume - Support and Troubleshooting

CMDB CI Deletion Jobs for High Volume Background A customer whose workloads primarily run on Kubernetes was generating huge volumes of ephemeral CIs (e.g., ~1M Docker containers/day + 300-1500 related software installations per container) that standard tools like TableCleaner and CMDB Data Manager were unable to keep up with.

To manage this volume of CI deletion, we created scheduled jobs that utilize either GlideTableCleaner or GlideMultipleDelete to remove batches of CIs. These jobs also bypass script engines, enabling us to delete at a high rate and clear out a backlog that included approximately 420M software install records.

CMDBCleaner CMDBCleaner allows cleanup jobs to be created that run on a schedule. These jobs can be split into multiple tasks for parallel processing. Each job runs at an interval and deletes a small batch of records. Both the batch size and the query for each job are configurable.

CMDBCleaner can be installed using the attached update set and is inactive by default.

It includes the following script includes for base functionality:

CMDBCleanerBase : Contains deletion functions and basic management of scheduled jobs. CMDBCIReferencedCheck : Determines whether a CI is currently referenced by other tables. The base class should be extended to create jobs specific to the table you want to clean. The extending class must implement the process function to define how the CIs are cleaned from the CMDB.

The script flow is as follows:

Obtain a batch of sys_ids to delete. Delete them and/or related records using either GlideTableCleaner (with cascades, no engines), GlideMultipleDelete (no cascades, no engines), or GlideRecord.deleteMultiple (with cascades, engines). Sample jobs:

CMDBCleanerK8SComponents : Uses GlideTableCleaner to cascade delete any cmdb_ci_kubernetes_component CIs. CMDBCleanerDockerContainers : Uses GlideTableCleaner to cascade delete any cmdb_ci_docker_container CIs. CMDBCleanerDockerContainersGMD : Uses GlideMultipleDelete to clean any cmdb_rel_ci records, cmdb_sam_sw_install records, and any unreferenced container CIs. Referenced ones are sent to GlideTableCleaner. Creating a new Cleaner To create a new cleaner, extend the CMDBCleanerBase class with a new script include.

Note that the table name is passed to the base class's init function. After this, you can set any hardcoded or property-driven variables for the cleaner, such as the batch size or the query to be used.

The process function must be created and should first call the getIds() function, which retrieves a batch of sys_ids from the defined table and query. After this, the exact process is up to you, but you should expect to use the deleteIdsTC , deleteIdsGMD , or deleteIdsGR functions at least once.

var CMDBCleanerK8SComponents = Class.create();

CMDBCleanerK8SComponents.prototype = Object.extendsObject(CMDBCleanerBase, {

initialize: function() {

var table = 'cmdb_ci_kubernetes_component';

CMDBCleanerBase.prototype.initialize.call(this, table);

this.addEncodedQuery(GlideProperties.get('cmdb.cleaner.k8s.query', 'install_status=100'));

this.setBatchSize(GlideProperties.getInt('cmdb.cleaner.k8s.batch_size', 100));

this.setDryRun(GlideProperties.getBoolean('cmdb.cleaner.k8s.dry_run', false));

},

process: function() {

this.logParams();

var ids = this.getIds();

if (ids.length == 0)

return;

this.deleteIdsTC(this.table, 'sys_id', ids);

this.logDuration();

},

type: 'CMDBCleanerK8SComponents'

}); Managing Jobs The cleaner would typically be run by a one or more scheduled jobs. If you plan to multi-thread the deletion, the the createJobs function can make this easier, for exmaple to create 4 jobs that run at 1 minute intervals you would run this:

var duration = new GlideDuration(60000); var cleaner = new CMDBCleanerK8SComponents(); cleaner.deleteJobs(); cleaner.createJobs(duration, 4);

The result would be 4 sysauto_script records that run at 1 min intervals, initially starting 15secs apart.

When using multiple jobs, the work is split by adding a query that filters by the first letter of the sys_id . For example, the job created above would split the work as follows:

CMDBCleanerK8SComponents_1 - sys_id BETWEEN 0@4 CMDBCleanerK8SComponents_2 - sys_id BETWEEN 4@8 CMDBCleanerK8SComponents_3 - sys_id BETWEEN 8@c CMDBCleanerK8SComponents_4 - sys_id BETWEEN c@g Note: The BETWEEN function for matching is inclusive on the left of the @ and exclusive on the right. For example, sys_id BETWEEN 0@4 only includes sys_ids 0, 1, 2, and 3.

Job Properties The jobs provided allow configuration of several properties, including the batch size, query, and whether it should be a dry run.

For example, the CMDBCleanerK8SComponents job includes the following properties:

cmdb.cleaner.k8s.query : An encoded query string. cmdb.cleaner.k8s.batch_size : The maximum number of records to delete on each run. cmdb.cleaner.k8s.dry_run : A flag that determines if the job should actually delete records or just simulate the deletion. To view currently set system properties navigate to:

All > system properties > All properties Search for cmdb.cleaner in Name column as shown below screenshot.

Logging Logs for the jobs can be viewed in the syslog by filtering for the source *** Script [CMDBCleaner] . The log level is controlled by the system property:

cmdb.cleaner.loglevel

Steps to Import and Commit Update Set Before you begin Role required: sys_admin/Global

Procedure Download the sys_remote_update_CMDBKubernetesCleanerV1.xml update set from this KB article. Elevate your privileges to the security_admin role if it is not already done. Navigate to All > System Update Sets > Retrieved Update Sets. Click the Import Update Set from XML link. Click Choose File and select the sys_remote_update_CMDBKubernetesCleanerV1.xml file. Click Upload. After uploading the file to the instance, click on the uploaded CMDBCleaner v1 update set. Click on the Preview option to ensure there are no errors while importing the update set. Commit option will be shown. Click on that to import the update set. This updateset will auto configure 2 sets of schedules jobs:

2 x CMDBCleanerK8SComponents to delete Absent CI's from the cmdb_ci_kubernetes_component table 2 x CMDBCleanerDockerContainers to delete Absent containers from the cmdb_ci_docker_container table These scheduled jobs are running at the interval of 1 min.

These scheduled jobs will clean up to a total of ~1.7M absent containers or kubernetes components per day, at a max rate of 600 CIs per job, per minute.

To view the scheduled jobs, navigate to:

All > System Definition > Scheduled Jobs Search for cmdbcleaner in Name column as shown below screenshot.