Mastering Data Management in ServiceNow: from tracking growth to efficient cleanup - Support and Troubleshooting

Mastering Data Management in ServiceNow: from tracking growth to efficient cleanup In this Knowledge Base article, we provide a holistic guide to managing your data in ServiceNow, covering everything from monitoring da tabase growth to optimizing large tables. Explore best practices for self-managed data cleanup, learn about the available tools and features, and gain insights on how to tackle your largest tables effectively. Arm yourself and team with the knowledge to make informed decisions about data management for smoother operations.

Note: Deleting large datasets may take weeks to months to process. This duration ensures a thorough and systematic removal, minimizing disruptions to your operations.

Note: Please ensure you login to the NowSupport Portal to view the more detailed documents linked within this KB Article.

Table of Contents Best Practices for Self-Managed Data Cleanup and Removal Best Practices for Data Removal Tracking Table Growth and Overall Database Footprint Available Data Management Products and Features Database Compaction – Reclaiming Table Free Space Delete Jobs Table Cleaner Data Archiving CMDB Data Manager Clones Scripted Methods Strategies for Managing Your Largest Tables sys_attachment/sys_attachment_doc sys_audit sys_audit_delete sys_email ar_* tables and sys_archive_log syslog* cmdb* sh* and sys_rollback* tables pa_scores* and pa_snapshots tables

Best Practices for Self-Managed Data Cleanup and Removal Maintaining a clean and organized ServiceNow environment is essential for optimizing system performance and ensuring the accuracy of your data. To help you manage your data effectively, we’ve outlined some best practices for self-managed data cleanup and removal.

ServiceNow Customer Support, in most instances, will not assist in removing data from a customer’s instance. Here's why and what you can do.

Integrity & Accountability: Directly intervening with data deletion could risk the integrity of a customer's unique setup and data. We prioritize the safety and consistency of your data. OOTB Features: ServiceNow offers out-of-the-box (OOTB) features that can assist in data cleanup. However, large datasets may take weeks to months to process. This duration ensures a thorough and systematic removal, minimizing disruptions to your operations. Self-Service Tools: We provide tools and documentation that empower users to manage and clean their data efficiently and safely. Recommendations: Before attempting a significant data cleanup: Testing: Test the cleanup process in a sub-production environment with a similar size dataset before executing in a live environment. Consult: Consider reaching out to a ServiceNow consultant or partner with experience in data management for guidance. Completion time: Keep in mind that the process of data removal may require a substantial amount of time, so plan accordingly. Large datasets may take weeks to months to process. Best Practices for Data Removal Below knowledge article link will outline the guidelines to ensure your environment remains efficient, secure, and compliant with your organization's data policies.

Best Practices for Data Removal

Tracking Table Growth and Overall Database Footprint To maintain optimal performance and ensure your ServiceNow environment is running efficiently, it is essential to monitor the growth of tables and the overall database footprint. Understanding and tracking these metrics will help you identify potential issues before they impact your system's performance or scalability.

Please follow below knowledge article link for key considerations for tracking table growth and your database footprint within ServiceNow instance.

Tracking Table Growth and Overall Database Footprint

Available Data Management Products and Features As you navigate the complexities of data management, knowing your options is invaluable. Below are the features and products you can utilize to manage and clean your data effectively in a self-service manner.

Database Compaction – Reclaiming Table Free Space To ensure your ServiceNow environment continues to perform at its best, it’s important to maintain the health of your database. Over time, as data is added, updated, and removed, the database may accumulate unused space. This can lead to inefficiencies and potentially impact system performance. Database compaction is a key process that helps optimize your instance by reclaiming unused space, improving overall system performance, and ensuring efficient storage management.

What is Database Compaction?

Database compaction is the process of reorganizing the database to remove any unused or fragmented space. This helps reduce the overall size of the database, enhances system speed, and ensures that your environment remains efficient and responsive.

Why is Database Compaction Important?

Improved Performance: As data accumulates over time, it can cause fragmentation, slowing down queries and processes. Compaction helps improve the speed and efficiency of your database operations.

Optimized Storage: By reclaiming unused space, compaction reduces the total database size, freeing up valuable storage resources and making better use of your system’s storage capacity.

System Health: Regular database compaction can prevent potential performance issues related to excessive fragmentation, helping ensure your instance operates smoothly.

Please follow below knowledge article link to understand more on Database Compaction. Database Compaction – Reclaiming Table Free Space

Delete Jobs Occasionally, you may need to delete data that is no longer needed. ServiceNow offers a streamlined process to manage and delete records safely, ensuring that your instance remains clean and free from unwanted data. By removing obsolete data, you help free up system resources, allowing for smoother performance of your ServiceNow instance.

Please check below knowledge article link for more information on delete jobs.

Delete Jobs

Table Cleaner The Table Cleaner is a powerful tool in ServiceNow that allows you to automate the cleanup of records in specific tables. This feature helps to remove outdated or unnecessary data from your ServiceNow instance, which in turn improves system performance, reduces clutter, and ensures data integrity. The Table Cleaner is a recurring job that operates hourly (by default) to remove outdated, expired, or unnecessary records from tables. By doing so, it helps keep tables from becoming overly large and ensures better query performance. Please follow below knowledge article link for more details on Table Cleaner functionality: Table Cleaner

Data Archiving As your ServiceNow instance grows, it's essential to manage the increasing volume of data to ensure both system performance and compliance with organizational policies. One effective way to manage this data is through Data Archiving. This process allows you to move older, less frequently accessed records into a separate storage location, reducing the load on your active tables while still maintaining access to important historical data.

What is Data Archiving?

Data archiving in ServiceNow is the process of moving older or infrequently used records from active tables to a set of archive tables. These archived records are still accessible if needed, but they no longer consume resources in your primary system, ensuring that the system operates efficiently.

Why Archive Data?

Optimize System Performance: Archiving helps reduce the size of active tables, which can improve system responsiveness and query performance by minimizing the amount of data that needs to be processed.

Improve Storage Management: By archiving outdated records, you free up valuable storage space in your active database, which can be better utilized for current, active records.

Ensure Compliance: Archiving allows you to retain historical data for compliance purposes while making sure that only relevant data remains in active use.

Please follow below knowledge article link for more details on data archiving.

Data Archiving

CMDB Data Manager The CMDB Data Manager in ServiceNow is an essential tool that helps you effectively manage and maintain the integrity of your Configuration Management Database (CMDB). The CMDB is a critical part of your ServiceNow environment, as it stores details about the Configuration Items (CIs) that are key to your IT services and operations. Ensuring that this data is accurate, up-to-date, and properly organized is vital for efficient service management and decision-making.

Please follow below knowledge article link for more details on CMDB Data Manager. CMDB Data Manager Clones A clone in ServiceNow is a full or partial duplication of your instance that includes everything from your configurations, scripts, data, and custom applications. Cloning helps you replicate your production environment into a non-production instance or to copy data between non-production instances for purposes such as testing, training, or troubleshooting without impacting live operations.

Please follow below knowledge article link for more details on clones. Clones

Scripted Methods For customers who are familiar with scripting within the ServiceNow platform, GlideRecord delete() and deleteMultiple() methods can be utilized as part of scheduled jobs, fix scripts, or from the Scripts - Background module. Scripted methods allow for increased flexibility, like running parallel jobs (limit to 4) on different sub-sets of the target dataset for improved rates of deletion.

Scripted Methods

Strategies for Managing Your Largest Tables This section provides insights into the most commonly large tables that frequently catch the attention of our customers. Whether you're dealing with data accumulation issues or simply looking to optimize your storage, these tips will help you effectively manage and streamline your largest tables.

sys_attachment/sys_attachment_doc In ServiceNow, managing attachments—such as files, images, or documents—is a common part of various workflows, from incident management to change requests. ServiceNow provides special tables that help store, organize, and manage these attachments efficiently. Two important tables in this process are sys_attachment and sys_attachment_doc.

What is sys_attachment table?

The sys_attachment table is the main record-keeping table for all attachments in ServiceNow. It holds the metadata associated with each attachment, including details like:

The table to which the attachment is linked (for example, an Incident, Change Request, or Knowledge Article). The name of the attached file. The size of the file. The attachment type (such as image, PDF, or document). What is sys_attachment_doc table?

The sys_attachment_doc table, on the other hand, stores the actual content of the attachment—the file data itself. This is where the binary file is stored after being uploaded to the system.

In simpler terms:

The sys_attachment table stores metadata (information about the attachment). The sys_attachment_doc table stores the binary content. To understand more on these 2 tables and how you can maintain data in them, please follow below knowledge article link:

sys_attachment/sys_attachment_doc

sys_audit The sys_audit table plays a vital role in tracking changes and ensuring that all updates to records are properly logged for auditing purposes. What is sys_audit table?

The sys_audit table in ServiceNow is a system table that records audit information whenever a change is made to a record in the system. It captures details about changes made to fields, including:

What was changed (e.g., field values, data) Who made the change (the user or system performing the modification) When the change occurred (date and time) Old and New values for the fields that were updated Which record was modified The sys_audit table helps create an audit trail of changes, allowing your organization to track record modifications for compliance, troubleshooting, and transparency purposes.

To understand more on how you can maintain data in sys_audit table, please follow below knowledge article link: sys_audit sys_audit_delete The sys_audit_delete table in ServiceNow is used to record the deletion of records that are being audited. This data can be used to track deletes and is used in part for the Delete Recovery feature (not rollback). To understand more on how you can maintain data in sys_audit_delete table, please follow below knowledge article link:

sys_audit_delete

sys_email Email communication plays a crucial role in managing incidents, requests, notifications, and other workflows. The sys_email table is a key component of the system that helps track and store all email-related activities, providing visibility into the communication sent and received within your ServiceNow instance.

Think of it as the "sent" and "inbox" folders combined in your regular email client, but for the ServiceNow system. The table logs important details about each email, including the content, sender, recipient, status, and processing information. It helps ensure that your organization has a comprehensive record of email communications, which can be valuable for tracking, auditing, and troubleshooting purposes. To understand more on how sys_email table works and how you can maintain data in this table, please check the below knowledge article: sys_email ar_* tables and sys_archive_log These tables are part of the data archiving product. The ar_* tables contain records that were moved from the active tables to the archived table and the sys_archive_log table contains the metadata plus the payload to restore the record if required. More details on how to manage the size of these tables, please check the below knowledge article:

ar_* tables and sys_archive_log

syslog* The syslog tables in ServiceNow are specialized system tables that store log entries related to system activity, performance metrics, errors, and warnings. These logs are generated automatically by the system during various operations, such as executing scripts, processing workflows, or handling integration requests. The syslog tables enable administrators and support teams to track events, errors, and other significant activities that occur within ServiceNow instance.

More details on how to manage the size of these tables, please check the below knowledge article:

syslog*

cmdb* More details on how to manage the size of these tables, please check the below knowledge article:

cmdb* sh* and sys_rollback* tables Rollback contexts in ServiceNow provide a powerful way to reverse certain actions and recover from unwanted changes. With rollback, you can undo specific actions such as patch upgrades, plugin activations, and background script executions. Additionally, it enables you to restore deleted records along with all related modifications, ensuring that critical data can be recovered when needed.

Please check the below knowledge article for more details on what these tables are how they can be maintained.

sh* and sys_rollback* tables

pa_scores* and pa_snapshots tables As your Performance Analytics scores and snapshots accumulate over time, they can increase in size, which may affect system performance and the accuracy of your data. To maintain optimal performance, it’s important to regularly clean up these scores and snapshots.

ServiceNow uses a scheduled job to remove outdated scores and snapshots. The Clean PA Collections job can be enabled and run daily. This job ensures that only a small amount of data is deleted each time, with no significant impact on system performance.

You can check the below knowledge article for more details:

pa_scores* and pa_snapshots tables