Agent Client Collector / DEX and Clones

Table of Contents Are ACCs cloned? Clone Profiles, Excludes and Preservers ACCs connected via MID Servers API Keys Extension Context records ACC Agents or DEX clients connected directly to the instance Configuration Files ACC CIs and records sys_class_path mismatches ACC Plugins (Sensu Assets) Pattern script files App/Plugin and Installed versions ACC-M - Monitoring / MetricBase ACC-L - Health Log Analytics When the clone source doesn't have HLA Instance vs. Occultus version incompatibility Are ACCs cloned? No.

All Agent Client Collector installs are set up to point to groups of MID Servers, which in turn are set up for specific instance URLs. ACCs and MID Servers will continue to connect to the same instance URL after a clone, even though that instance may now be a very different configuration.

ACCs have a limitation of only 1 ACC install per host, so even manually cloning ACC installs for different instances won't work. An Agent install can manually be reconfigured to communicate with a different MID Server/Instance URL instead, and effectively switch to a different instance, but cannot be set with URLs for different instances at the same time.

Clone Profiles, Excludes and Preservers By the cGTM release, ACC-F had these exclude/preserve settings (added by PRB1408738), and are all included in the System Profile:

Excluded: sn_agent_cmdb_ci_agent sn_agent_mid_info sn_agent_request_checks sn_agent_ci_extended_info sn_agent_test_result sn_agent_request sn_agent_policy_clients ecc_agent_ip_address (from ACC-F v2.3.01 PRB1437462) Preserved: ecc_agent_ip_address (from ACC-F v2.3.01 PRB1437462) Known problems:

KB2032995 / PRB1866874 ACC related tables specific to Agents of the clone source instance should be excluded in clones See below for MID Server Extension Context settings.

ACCs connected via MID Servers Agent Client Collector installs connect to the instance via MID Servers, and so all potential clone issues for MID Server do also need taking into account when considering setting up Clones for instances with ACC:

KB0786475 MID Servers and Clones Each Agent Client Collector installation communicates to a MID Web Server Extension running on a specific MID Server, and therefore specific instance name. The ACC Websocket Endpoint and MID Web Server Contexts need to still be there, with the same ports and credentials settings after a clone, or the Agent Client Collectors will not be able to communicate with the MID Server and instance any more.

Excluded: ecc_agent_ip_address (from ACC-F v2.3.01 PRB1437462) Preserved: ecc_agent_ip_address (from ACC-F v2.3.01 PRB1437462) See below for MID Server Extension Context settings.

API Keys Table mid_webserver_api_key_credentials, added in San Diego release, extends base table discovery_credentials, which is excluded and preserved.

It's documented that in general that if a parent table is excluded then all of its children will be excluded as well. However PRB1578751 was opened because these were not being preserved, however this turned out to be a side effect of a clone engine issue that was fixed in June 2020 by PRB1403259.

Before then, table ecc_agent_webserver_api_key was used, which has an exclude/preserver.

There should not be a problem with this now.

Solution:

Backing up the API Keys, especially before a clone, would be a good idea. They can then be re-imported, or added again afterwards.

Extension Context records These extensions need to be preserved/excluded in clones:

MID Web Server ACC Websocket Endpoint plus any others related to Event Management/Operational Intelligence etc. These are now excluded/preserved:

Excluded: ecc_agent_ext_context_webserver sn_agent_ext_context ecc_agent_ext_context_trap (from Vancouver PRB1628241) ecc_agent_ext_context_vcenter (from Vancouver PRB1628241) ecc_agent_ext_context_metric (in Extended Profile from Vancouver PRB1628236) ecc_agent_ext_context_event (in Extended Profile from Vancouver PRB1628223) Preserved: ecc_agent_ext_context_webserver ecc_agent_ext_context sn_agent_ext_context ecc_agent_ext_context_trap (from Vancouver PRB1628241) ecc_agent_ext_context_vcenter (from Vancouver PRB1628241) ecc_agent_ext_context_metric (in Extended Profile from Vancouver PRB1628236) ecc_agent_ext_context_event (in Extended Profile from Vancouver PRB1628223) The instance clone automation hasn't been very good at excluding or preserving individual Classes within a TPC extended table hierarchy. This results in corrupted records that can't be opened and run properly by the mid server, and then duplicate records getting created that might appear to be the only record.

You can get an overview of all extension in the ecc_agent_ext_context table, because everything is a child table of that.

If a clone has corrupted these records, some record may not open with the error "Record Not Found".

Known problems with the out-of-box preservers/excludes:

PRB1408738 Agent Client Collector tables are missing Clone Exclude/Preserver settings - Fixed since ACC- F 2.1.41. KB1212634 / PRB1628241 Clone Excludes/Preservers are missing for SNMP Trap and vCenter Event based Discovery MID Server extension contexts (ecc_agent_ext_context_trap / ecc_agent_ext_context_vcenter) KB1212633 / PRB1628236 Clone Excludes/Preservers are missing for Metric Intelligence MID Server extension contexts (ecc_agent_ext_context_metric) KB1212632 / PRB1628223 Clone Excludes/Preservers are missing for Event Management MID Server extension contexts (ecc_agent_ext_context_event / eif_listener_context) These are all fixed in Vancouver, however there may still be Clone engine known problems that cause this when not every table in the extended table hierarchy is exclude and preserved in the clone.

Solution:

To repair bad records, please use this KB article: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0716609

To find the records, set the script up with: findOrphans('ecc_agent_ext_context', null, false);

Then to delete them use: findOrphans('ecc_agent_ext_context', null, false);

You may then need to set up the MID Server extensions/collectors/listeners again.

Prevention:

To prevent this in future, you will need to add Clone excludes and preservers for ecc_agent_ext_context and all the tables that extend it.

You can see a list of what child tables you have in your instance using this list filter: https:// .service-now.com/sys_db_object_list.do?sysparm_query=super_class.label%3DMID%20Server%20Extension%20Context%5EORsuper_class.super_class.label%3DMID%20Server%20Extension%20Context&sysparm_first_row=1&sysparm_view=

ACC Agents or DEX clients connected directly to the instance These bypass the MID Server so are less of a problem. These agents will be authenticating directly with the instance, so the User records, and any mTLS certificates will need preserving in the clone.

The following tables were preserved/excluded prior to the cGTM release, but should not be (PRB1728925). Doing this causes Fenix to fail to process the data:

DEX Metric Definition dex_metric_definition DEX Metric Processing Rule dex_metric_processing_rule DEX Metric Source dex_metric_source DEX Metric Tag dex_metric_tag DEX Metric Tag M2M dex_metric_tag_m2m DEX Source Metric Configuration dex_source_metric_config DEX Source Tag M2M dex_source_tag_m2m

Configuration Files Table sn_agent_configuration_file is cloned, and should be, which means Agents connecting to the target instance should now use the config files copied from the source instance.

Known Problems:

PRB1734261 config files meta data is not sent to agent after instance clone This is fixed in ACC-F 3.5.3 ACC CIs and records 2 main tables record the ACC details:

Agent Client Collectors [sn_agent_cmdb_ci_agent] The main record. This is a CMDB CI Class. If you are referencing an ACC from another record, this is the one you reference. Agent Client Collector Info [sn_agent_ci_extended_info] Additional fields. Each of the above records has one of these too, and references it. Most of the status columns seen on a list of Agents are actually dot-walked from this record. In effect this pair of tables are 2 halves of the same record. Unfortunately clones cannot reliably exclude and also preserve single CI classes, especially for complex TPP extended tables like the CMDB.

This means the records of the Agent installs that talk to the target instance, or via the target instance's MID Servers, will be lost in clones. The records copied from the target will be irrelevant to this instance. They should get recreated automatically though as the agents check in after the clone.

sys_class_path mismatches When any app or plugin adds tables/classes to the CMDB extended table, a sys_class_path value is randomly created for that table, and stored in the sys_db_object record. When plugins like this are installed independently on different instances, the sys_class_path values will be different.

That means records in the clone target instance, that are preserved, will be set with what is now an invalid sys_class_path value after the clone. You will still see those records in a top level listing of the cmdb, but will not be able to open them, with 'Record not found' errors.

The related sn_agent_ci_extended_info record of the original CI records also gets deleted.

The result is that that new duplicate CI records get automatically created when the pre-existing ACC installs start communicating with the instance after the clone.

PRB1555641 was opened for ACC, but is caused by the more general TPP clone problem PRB1411998, which for the moment is deemed 'working as expected'. TPP extended tables, like the CMDB, don't support Preservers, at least until a clone means the 2 instances are now in sync for the class path of each class in the CMDB.

Solution:

If you have corrupt records that can't be opened or deleted, remove the clone preservers for sn_agent_ci_extended_info and sn_agent_cmdb_ci_agent, and re-clone. This will delete the corrupt records left preserved from the previous clone.

Don't preserve either sn_agent_extended_info or sn_agent_cmdb_ci_agent for the first clone after installing ACC-F on both instances. When Agents check back in after the clone, these will be re-created anew.

From then on, it should be possible to add Preservers back again, because the sys_class_path values will now be the same in both instances.

Since ACC-F cGTM 202009, agent_extended_info and sn_agent_cmdb_ci_agent are excluded from clones, and not preserved, so these records will not be copied from the source (which would be pointless, as those Agents are the source's ones). , or left on the target (to avoid this problem).

Install ACC-F on the clone source instance first. After the clone, the target instance will have the same sys_class_path value for the sn_agent_cmdb_ci_agent class, so when ACCs are installed for the target instance, they will not have problems in future clones.

See the Known Error article for more details: TBC

ACC Plugins (Sensu Assets) ACC Plugins (aka Sensu Assets) are 'code' and so should be copied in a clone. All records in sn_agent_asset, and their .tar.gz file attachments should be copied over in the clone, replacing any records that were on the target.

However these were are being treated as large attachments, and deleted.

Since March 2022 the clone engine knows to copy these attachments in clones so you should not longer see this problem.

See the Known Error article for more details: KB1002379 / PRB1550798 Clones that exclude large attachments break Agent Client Collector, by deleting the tar/gz files attached to ACC Plugin records in sn_agent_asset

Solution:

Repair all the Agent Client Collector plugins to get the attachments back. Any custom ACC Plugins can be imported as XML from the clone source. XML export of the ACC Plugin record will also include the attachment data.

Pattern script files During agentless pattern execution, when we need to upload a script to a target device, we use files that are stored as attachments to records on "sa_uploaded_file" table. Since PRB1507034 fixed the clone engine in July 2021, those attachments should always get cloned, even if "Exclude Attachments" is true in the clone setup.

Pattern execution via ACC can not use the "putFile" method, however some (not all) patterns have now had the files that would have been used for putFile placed as assets into a ACC Plugin (Asset) called pattern-execution. For example, the Oracle GLAS pattern uses these attachments for SQL scripts.

But, the MID will still first check the "sa_uploaded_file" table and if the records or attachments are missing there, then the MID will not direct the agent to use the file from its own assets cache folder.

App/Plugin and Installed versions A clone might downgrade plugins/apps, and potentially remove plugins/apps and the tables that go with them.

A clone target instance that is on a later version of CC apps/plugins, and has ACCs installed on servers at those later versions, could be overwritten by a source instance on a previous ACC version. Future version installs will now be linked with an instance that is on an earlier version, and code and checks may be incompatible.

A clone source instance that doesn't even have ACC apps installed, won't have the ACC code or tables, and so nor will the clone target instance after the clone. Those tables, the records they contain, and the code to go with them is deleted as part of the clone. The MID Server web server extension context and websocket lister code will no longer exist, and so although the ACC installs will continue to try to communicate with their MID Server, the MID Server will not do anything with those requests. Those orphaned ACCs may need re-installing once ACC is set up in the instance again after the clone.

Solution:

Install and upgrade ACC related plugins on the source instance first, to match what is on the target instance, before cloning. ACC does not need to be set up or in use on the clone source instance, just present.

ACC-M - Monitoring / MetricBase Event type ACC checks will populate the Event Management SQL em_event table, but Metric type checks for Metric Intelligence will populate a time-series database called MetricBase (aka Clotho), which is a separate node/instance to the normal Glide appnode and sql database nodes.

MetricBase content is cloned from Source to Target.

If Target does not have MetricBase configured or installed, then the MetricBase plug-in will have to be installed (or re-installed) after clone to access MetricBase content.

ServiceNow support engineers will have access to these internal articles to aid with troubleshooting after clones:

KB0678880 Internal Support Documentation for MetricBase KB1582884 Plugins that will be reinstalled by the Clone workflow under specific conditions Cloning supports MetricBase after Jakarta Patch 7, when PRB1159519 added exclude/preserver for sys_clotho_config.

ACC-L - Health Log Analytics HLA is built from several integrated components, which means there is more than just the code in the Glide application nodes, and data in the SQL database. HLA Plugin & Scoped App, running in the Glide application node Occultus Instance Elastic Search data Metric based data During the clone process the data from the source instance, from all the relevant components listed above is copied to the target instance.

The cloning process provisions a new ElasticSearch, Metric Base and if needed an Occultus instance, it also updates the relevant tables with the new configurations.

For more information, see:

KB0867530 System Clone - Health Log Analytics | HLA Quebec ServiceNow support engineers will have access to these internal articles to aid with troubleshooting after clones:

KB1002293 HLA Backup Procedure - Supporting Customer Clone KB1273656 Work Instruction | How to troubleshoot HLA issues in target instances after cloning KB1119941 Work Instruction | Health Log Analytics (Loom) - Manual Clone steps There are some issues with HLA in general due to clones, which are not ACC-L specific but can mean the Agents or MID Server extension contexts error. Include but not limited to:

When the clone source doesn't have HLA If the clone source does not have Health Log Analytics plugin installed, then after the clone the target will also not have the plugin installed. You cannot 'preserve' apps/plugins. However the target's Occultus instance will still remain, and the Service records will have been preserved.

In this situation you could either

Install HLA on the target or, decommission the Occultus node Instance vs. Occultus version incompatibility After a clone, where the target instance version changes, the existing Occultus node version may now be incompatible with the instance version. The Occultus node may need upgrading or downgrading to match.