Archiving | General OverviewSummaryThe archive application moves data that is no longer needed every day from primary tables to a set of archive tables. Archival tables are flattened tables ie. Base and child tables are flattened into a single table and Reference fields are converted to strings. Plugin: Data Archival plugin is active in an out of box instance by default. Plugin details: Name: Data Archiving ID: com.glide.auxdb Link to the documentation: Data archiving Archival Rules: You can create 'Archival Rules' to configure what data to archive. In the application there is a module 'Archive Rules' where this can be set up and the name of the rule and the table name are required fields. It is recommended to add a condition to the rule. Once, the rule is created it will be set to an Inactive state, Make sure to set active to true. Navigation: System Archiving > Archive Rules There can be only one archival rule that can be set up per table whether it is active or inactive. We don't allow an archive rule to be defined for any of the SYS tables.The exceptions are listed in property glide.ui.permitted_tables. You cannot archive tables that have table rotation enabled. Activating an Archive Rule Immediately If you do not want to wait for the scheduled job to run the archive rule, you can manually start the archive rule. Navigate to System Archiving ➔ Archive Rules.Select the archive rule you want to run.Click the Run Archive Now related link. Archiver Job: Once the archival rules are set up and there are archival rules which are active, the archiver job will process these rules and archive the records. The Archive scheduled job is set up as a one-hour repeat interval job. It calls a class that hands the thread off to a background thread called glide.db.archiver. In order to prevent the archive process from consuming too many system resources, the below archive properties can be used control how many records the archive rule processes at one interval. However, we do not recommend to change these properties. By changing the above properties, the Archiver job can exceed the 1hr duration and we can have multiple archive jobs executing. The archival process happens in this order: Load N records that meet the archival rule criteria (where N is the batch size property)Insert into ar_* tableDelete the record from the original tableCreate a sys_archive_log entry When a record is archived, an entry is made in Archive Log(sys_archive_log) table and the xml of the record is saved as the payload. There are a few columns that are helpful in debugging - Archival run, Archive, Restored,ID and From table. Archival run - This will give you the details, of which particular archive run archived this record Archive - The archival rule that met the condition Restored - This field will be blank if the record has not been restored. If the records has been restored, this field will contain the date and time the record was restored From table - The table from which the record was removed from The Archive Log will not be cleaned and we do not recommend to delete records from this table. Deleting an archive log entry prevents you from restoring the data for the archived records. The presence of an entry in Archive Log is all that the system needs to know that the record has once been archived (whether or not it was restored), in which case it is skipped. i.e. The archiving engine will not re-archive something that has been restored. Related records and attachments can be archived as well, as in the archive rule you can specify what related records you want to archive. If you restore the original record, then the related records needs to be restored as well(this is not automatic, as the related records tables are independent after archiving). Archive a record(s) Manually: You can manually archive each record by navigating to each form view and clicking "Archive Record" related link. You can manually archive multiple or a single record from the list view by selecting the list action "Archive Record". Link to the documentation for: Lists in the classic environment Record Estimate: You can determine the number of records that will be archived in the next run of the job for each of the archival rules. This can be achieved by clicking on the UI action "Record Estimate" for each of the archival rules. Once the UI action is clicked, there are 2 fields in the form that will be populated - Record estimate and Estimate Date. Record estimate which provides the estimated records that will be archived in the next run. Estimate Date will be updated with the current date and time when the estimated run was executed. Archive Destroy Rule: Archive Destroy rules delete archived data after a specified period of time. The Archive Destroyer batch process runs the Archive Destroy rules on archived data. You can configure some parameters of the batch process. Navigation: System Archiving > Archive Destroy Rules Create a scheduled job for each destruction rule you want to execute with the following script (Note: Ensure to test in sub-production instance first!):new GlideArchiver().archiveDestroyByRule(xxxx);where xxxx is the sys_id of the destroy rule. Link to the documentation: Create a destroy rule Archive Diagnostic Page: Link to access the UI page: https://<your-instance>.service-now.com/ui_page.do?sys_id=71681edc9f3120007aaa207c7f4bcc2b This page provided details of the number of archival rules that are active and the total records that were archived and the last run of the archive job. Related LinksData ArchivingBlog: Data Archival Walkthrough Known Problems: PRB608048 - Archive threads can run in parallel on separate nodes for long running archive processes, degrading performance and adding noise errors to the log. PRB1296280/KB0695364 - It is not possible to archive all cmdb_rel_ci record when CI records are Archived, leading to CompactRelation errors when the CMDB Identification Engine fails to consider that a CI may be Archived and throws an error, causing major syslog table growth PRB1164902/KB0780321 - Archive Destroy rule does not work when archived record is from a scoped application PRB592993/KB0791832 - Archiving throughput impeded by too many restored records FAQ: Q. Can Archival be enabled for rotated tables?No, Archival cannot be enabled for rotated tables. Q. Can Related records be archived?Yes, use the Archive Related Records related list to add related records to the archive rule. Q. Can all users view the archived records?By default, archive tables use the ACLs for the unarchived table of the same name. For example, the archived Incident [ar_incident] table uses the ACLs defined for the unarchived Incident [incident] table. Refer to Data archiving for more details. Q. Can Destroy rules be used to delete records from Archive Log?Destroy rules can be used to delete record from the archive tables only (ar*) Q. Can Table cleaner be used to delete from Archive Log?If you delete a record from Archive Log that has not been destroyed yet, you will lose the ability to restore it.