CMDB Baseline life-cycle best practices and Diff Formatter troubleshooting - Support and Troubleshooting

CMDB Baseline life-cycle best practices and Diff Formatter troubleshooting Summary Note: CMDB Baseline is a deprecated feature , and customers are discouraged from using it. If you are interested in stopping using it, see KB1262214 CMDB Baseline database footprint and disk space usage explained which also discusses deleting everything and removing the formatter from your form layouts.

Table of Contents Planning: Decide which CI Classes will be included in the Baselines Decide if all CIs in those Classes are to have Baselines Decide how regularly you will create new Baselines Decide how long you will keep old Baselines Implementation: Configure Auditing for the relevant CMDB Tables and Fields Implement the Scheduled Script to Create Baselines Implement the Deletion Policy Add the Formatter to the CI forms Troubleshooting Form won't load and times out Not all changes load No Baseline is shown A change I expected to see is not shown Known limitations / Problems Planning: Decide which CI Classes will be included in the Baselines Decide which Tables/Classes you need to have CMDB Baselines for. There are over 700 CI classes in the CMDB extended table, and most wouldn't need baselines. What are you interested in using Baselines with? Just the main parent CIs for Hardware? Only Servers and computers? You will usually end up with a short list containing only a few branches of the CMDB.

You would then usually end up with a Baseline created for each branch of the CMDB you are interested in. The Table specified when creating the Baseline is the top level table of the extended table branch, and all sub-classes in that branch will also be included, unless filtered out by the condition. Several smaller Baselines with simpler conditions is a lot easier to set up than one big baseline for cmdb_ci.

Note: You cannot configure which fields and related records are included in the baseline. Only which CIs are included.

Decide if all CIs in those Classes are to have Baselines You should further limit the conditions to filter out CIs where the Status is Retired, or any other attribute values which would mean a baseline would be pointless.

Decide how regularly you will create new Baselines The CMDB Baseline Diff formatter is not a full Audit History list. It is a list of changes to attributes, CMDB relationships, and related records, since the Baseline was recorded, for the purpose of quickly seeing what recent changes specific to the CI and it's related CIs have been made. For a full audit history, you should use the History - List feature instead.

A CMDB Baseline contains the snapshot of only the CIs that existed at the time the baseline was originally created . There is no way to add CIs to an existing baseline later, or add baseline entries for CIs created after creating the Baseline. Within a month or two baselines become unusable for a significant proportions of CIs.

You will need to regularly create new baselines, and how quickly the baselines become useless by having too many CIs missing will determine how regularly you do this. Monthly is normal, and weekly less so. Daily baselines of a significant part of the CMDB is likely to cause performance problems while creating the baseline due to the load on the instance and database. Come up with a sensible period for your own needs.

Decide how long you will keep old Baselines Baselines become useless over time, as too many of the in-play CIs are missing from them, and due to the time window of changes to display getting longer, too many changes are shown and the forms become unusable.

Changes older than a few months are usually not relevant to the work you are doing with the CI, and you always have the option of looking at History - List to go back further.

You will need a deletion strategy, as the feature does not have any Table Cleaner predefined ( PRB1283806 ). Database query performance can be severely impacted by the data from CMDB baselines. One customer who triggered the creation of baselines via business rules on changes, with no deletion strategy, ended up with a 3.4TB cmdb_baseline_entry table with 66 million records, which accounted for over 90% of the whole instance database disk space, including logs and attachments!

Keeping 1 or 2 old baselines is all you usually need, in case you want to look at older changes.

Implementation: Configure Auditing for the relevant CMDB Tables and Fields CMDB Baseline won't work without first turning on Auditing for each CI class you will create baselines for. Data on changes will only be available from then on. The Baseline diff formatter on the form is based on the snapshot created as the Baseline is initially recorded, and then all changes audited since.

There is no way to exclude fields in the formatter , and so the only way to remove unnecessary noise from the formatter is to prevent them being audited in the first place. Turn off auditing for any fields that you don't need or want to audit . Fields like "Last Discovered" don't need to be audited. Calculated fields , such as display names where the value is made up of a concatenation of other audited fields, usually don't need auditing.

To further reduce noise, turn off auditing for any fields that have regularly flapping values, or regularly changing values. e.g. free Disk Space updated daily by discovery. There may also be some known problems with Discovery causing flapping values that may make the formatter hard to use, such as IP Address ( PRB1380126 ), which you could try to fix by upgrading or implementing a workaround.

Flapping could also be caused by incorrectly set up Data Precedence Rules in the Identification and Reconciliation Engine . e.g. RAM values flapping daily between a Discovered value and a value known to be rounded differently from the SCCM import. That can be prevented by not allowing the SCCM Data Source to overwrite a value set by Discovery Data Source. (Upgrading the integration plugins to the post-Orlando Store versions, which do fully implement IRE and Data Precedence Rules may be required.)

Changes to Relationships are also listed, and usually account for most of the changes displayed. See this KB article for a way of turning off auditing of relationships completely if you only want attribute changes displayed in the Baseline Diff formatter: KB0817973 Turning off Audit on CMDB Relations

Implement the Scheduled Script to Create Baselines When a baseline record is manually inserted, the "SNC Create Baseline" business rule runs to create the baseline entries for each CI included in the table/conditions. A script can also simply insert a record.

Baselines can be created manually, but creation should be done with a scheduled script, running every month, week or perhaps 30 days. 3 values are required - name, table, condition - and the easiest way to get those values is to set up a CMDB list with only the records you want baselined included.

For example, We make up the "Name", but I recommend starting it with the Date of creation of the Baseline, e.g. "2020-06-22 Windows Computers" Open a list of the "Table", e.g. Computer [cmdb_ci_computer], with some filter conditions to limit the sub-classes to Windows-specific ones, and also filtering out Absent/Retired Status. Right clicking and Copying the Query provides you with the "Condition".

A simple script to insert a record with those 3 values is:

var baselineGr = new GlideRecord('cmdb_baseline'); // set up a GlideRecord object for the CMDB Baseline table baselineGr.initialize(); // initialize a new blank record. baselineGr.name = new GlideDateTime().getDate() + ' Windows Computers'; // uses the current date at the time the script runs baselineGr.table = 'cmdb_ci_computer'; baselineGr.condition = 'sys_class_name=cmdb_ci_computer^ORsys_class_name=cmdb_ci_win_server^ORsys_class_name=cmdb_ci_hyper_v_server^install_statusNOT IN100,7' baselineGr.insert(); // Insert the record, which will trigger the "SNC Create Baseline" business rule You can then set up a Scheduled Script to run that periodically, perhaps every 30 days.

Certain baselines could be created more regularly, and deleted later, according to your requirements, which is another reason for splitting different CMDB branches into different baselines.

Implement the Deletion Policy The simplest method is to use a Table Cleanup [sys_auto_flush] job, to delete all baselines older than a certain age, such as at 90 days. This would be in line with how often you create the baseline and how many old ones you are keeping.

See the Table cleaner section of the Data management docs.

System Maintenance -> Table Cleanup New Fill in the fields as below, and Submit Tablename: CMDB Baseline [cmdb_baseline] Matchfield: sys_created_on Age in seconds: 7776000 (for 90 days) Active: true Cascade delete: true (in order to also delete the cmdb_baseline_entry child records for each CI) Conditions: (Leave empty)

Add the Formatter to the CI forms For CI class and form View: Right clich the header -> Configure -> Form Layout Select the section Move "CMDB Baseline Diff" to the selected side, at the position you want Save The performance impact on form load times due to having the formatter on the form is noticeable. Only add the formatter to the forms of CI classes that you have created baselines for. You may also consider adding the formatter to the Form View of a particular role of user only.

Troubleshooting Form won't load and times out Temporarily removing the CMDB Baseline Diff formatter from the form layout will allow you to confirm if the formatter is the cause of the form not loading. Confirm if all CIs, perhaps of the same class, all have the problem, or if only some specific CIs do. Specific CIs or Classes tell you it is audit data specific to that CI or Class that is causing the problem. With the formatter removed from the form, try loading the History - List. Does that even load? Have there been thousands of updates? By who? which fields? when? How old is the Baseline that it is defaulting to, or that has been selected? If it was created more than a few months ago, it is too old to still be useful and should probably be deleted and replaced. A vast number of audited updates is usually the cause, either because the date range going back to the time of the baseline creation is too big, or because the audit history has a huge number of changes in it. This code may fail to finish querying the database, compiling the data, or finish rendering the list, if there is more audit data that it can cope with. To resolve that, you'd have to figure out:

What has been doing the updates, and which fields? May be imports, Discovery, business rules, or practically anything. This technique may help identify the code if the History - List doesn't. Should those updates have been happening, and can they be stopped? Is code doing updates when nothing has changed? Are Data Precedence Rules set up correctly to prevent flapping? Should those fields have been audited, and can they be set to not be audited in future? Might ServiceNow tech Support need getting involved to clear out some existing audit data? For obvious reasons, customers cannot do this themselves. Not all changes load The com.cmdb.baseline.max_changes system property limits the number of relationships and changes that appear in a baseline diff section on a CI form ( PRB1283753 ). You can try increasing that value, which is 10 by default, but the reason the cap exists is due to many performance issues in the past. If you have set it too high, you may have to lower it to get the form to load. The form isn't particularly usable if too many changes are shown, so it's better to reduce that is shown by regularly replacing your Baselines, and keeping an eye on what is updating the records and whether those updates need to be audited. No Baseline is shown If no Baseline was ever created for the CI, then no Baselines will be listed in the dropdown in the formatter.

You may have a baseline where the table/condition should cover the CI, but the CI may have been inserted since the Baseline creation.

A change I expected to see is not shown If the change is not in these tables it will not be shown. History - List takes its data from the same tables, so you will probably also not see the change in there.:

Sys Audit [sys_audit] Audit Deleted Record [sys_audit_delete] Audit Relationship Change [sys_audit_relation] Was Auditing only turned on recently for the relevant field or Table? Turning on auditing starts recording from then on only.

If the change pre-dates the creation date of the Baseline, it is not supposed to be shown.

Known limitations / Problems PRB1283806 CMDB Baseline: cmdb_baseline_entry table can grow to a huge size causing instance and database performance and disk space issues PRB1611377 CMDB Baseline creation job causing app node out of memory when a CI has a huge number of records referencing it PRB591082 CMDB Baselines allow baselining non CMDB tables even though the data cannot be displayed PRB1476641 When Baseline has a large number of records to populate (compute), it will slow down the loading of the ci's form Apart from PRB1611377, none of these are likely to be fixed.