DMTableCleaner | General Overview - Support and Troubleshooting

DMTableCleaner | General Overview Summary .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } In this Knowledge Base article, we provide an overview on DMTableCleaner and an operational guide.

Note: Deleting large datasets may take weeks to months to process. This duration ensures a thorough and systematic removal, minimizing disruptions to your operations.

Note: Please ensure you login to the NowSupport Portal to view the more detailed documents linked within this KB Article.

Table of Contents:

Introduction DMJobs Framework DMTableCleaner and Key Benefits How DMTableCleaner Works Navigating DMTableCleaner Job Producer/Consumer Configuration Error Codes & Warnings Introduction:

The increasing volume of data presents challenges for data management and cleanups. Cleaning data might be slower than the rate of data becoming eligible for cleanup. The DMTableCleaner, built upon the new Data Management (DM) framework, is designed to address present-day data housekeeping. It replaces legacy single-threaded cleaning mechanisms with a multi-threaded architecture, ensuring faster, safer, and more reliable data deletions.

DMJOBs Framework:

The DM (Data Management Framework) job is installed OOTB as part of net new plugin “DMJob Plugin”. The DMJob follows the producer and consumer pattern (Similar to Archiving job which is already using this pattern today). This DMJob framework is newly built to onboard Data management jobs using DM framework. The DMJob is configured in sys_trigger with name “DMScheduler” and is scheduled with repeat interval to run every one hour. 3 additional tables created with this DM plugin: sys_dm_job: Stores configuration details for each DMJob instance the framework can execute. sys_dm_run: Records every execution ("run") initiated by the producer job, For Example, each DMTableCleaner run typically corresponding to an active table cleaner rule. sys_dm_chunk: Contains the data "chunks" generated for each sys_dm_run, these are processed by consumer jobs. DMTableCleaner and Benefits:

The DMTableCleaner is a multi-threaded table cleanup solution utilizing the DMJob framework. It offers a distinct improvement over the legacy single threaded table cleaner by allowing multiple producers and consumers to operate in parallel. This configuration enables administrators to configure the degree of concurrency to the specific data growth and faster data cleanup needs of your instances. Key Benefits:

Parallel Processing: Multiple producers and consumers ensure that timeouts in one thread do not halt the overall cleanup process. Scalability: Admins can configure producers and consumers as needed to accelerate cleaning, especially in large-scale environments. Cost Efficiency: Quicker data cleanup translates to lower disk space usage and reduced costs. Resilience: If any producer or consumer encounters a timeout or error, others continue processing unaffected. How DMTableCleaner Works:

By default, from the Yokohama release, The DMScheduler in sys_trigger kicks off every hour. It goes through all active DMJobs (sys_dm_job), one of which is typically the DMTableCleaner. For each active table cleaner (sys_auto_flush) rule, a corresponding "sys_dm_run" is generated. The producer logic then creates system triggers named according to the 'producer_class' column in sys_dm_job—to manage the number of producer threads, based on the 'max_producers' value. Similarly, consumers are spun up with triggers named after the 'consumer_class' column and controlled by 'max_consumers'. A producer will grab any available DM run not taken already and will produce the DM chunks(sys_dm_chunk) for that DM Run. Consumers pick up available chunks and execute the data deletion from the appropriate tables. Producers and consumers operate in parallel, maximizing throughput and minimizing job duration. Once all the DM Chunks are processed, the DMJob marks the DM Run state to ‘complete’ Navigating DMTableCleaner Job:

The DMTableCleaner is enabled by default replacing the single threaded Table cleaner from Yokohoma release. However, in any case it can be enabled/disabled by following steps: Navigate to https:// .service-now.com/sys_dm_job_list.do Locate and open the DMTableCleaner Job. Toggle the active checkbox to enable or disable.

Producer/Consumer Configuration:

Fine-tuning the number of producers and consumers is straightforward. Within the DMTableCleaner job form, admins can set ‘max_producers’ and ‘max_consumers’ fields to scale up or down. Increasing these values allows for greater parallel processing, which is particularly important for environments with rapid data growth or exceptionally large tables to clean. Please note that customers are advised not to scale the number of Producers beyond 1, and Consumers beyond 3 , without first consulting ServiceNow support.

Error Codes & Warnings:

The “sys_dm_run” table stores the status of each run in ‘state’ column. Ready: Initial state when a run is created. Processing: When a consumer is actively consuming chunks for the run. Complete: When the run finishes successfully without errors. Error: Any error encountered during chunk processing or data cleanup. Timeout: When a run exceeds its allowed duration and is halted. Related Links .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: #000000; } span { font-size: 12pt; font-family: Lato; color: #000000; } h2 { font-size: 24pt; font-family: Lato; color: black; } h3 { font-size: 18pt; font-family: Lato; color: black; } h4 { font-size: 14pt; font-family: Lato; color: black; } a { font-size: 12pt; font-family: Lato; color: #00718F; } a:hover { font-size: 12pt; color: #024F69; } a:target { font-size: 12pt; color: #032D42; } a:visited { font-size: 12pt; color: #00718f; } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: block; max-width: ; width: auto; height: auto; } } Mastering Data Management in ServiceNow Manage table cleaner resource usage