Agent Client Collector and Clones

Table of Contents Are ACCs cloned? ACCs connect via MID Servers App/Plugin and Installed versions ACC CIs and records sys_class_path Associated records API Keys ACC Plugins (Sensu Assets) Extension Context records Are ACCs cloned? No. All Agent Client Collector installs are set up to point to groups of MID Servers, which in turn are set up for specific instance URLs. ACCs and MID Servers will continue to connect to the same instance URL after a clone, even though that instance may now be a very different configuration.

ACCs have a limitation of only 1 ACC install per host, so even manually cloning ACC installs for different instances won't work.

ACCs connect via MID Servers Agent Client Collector installs connect to the instance via MID Servers, and so all potential clone issues for MID Server do also need taking into account when considering setting up Clones for instances with ACC:

KB0786475 MID Servers and Clones Each Agent Client Collector installation communicates to a MID Web Server Extension running on a specific MID Server., and therefore specific instance name The ACC Websocket Endpoint and MID Web Server Contexts need to still be there, with the same ports and credentials settings after a clone, or the Agent Client Collectors will not be able to communicate with the MID Server and instance any more.

By ACC-F v2.1.41 (cGTM/202009) these exclude/preserve settings were included (PRB1408738).

App/Plugin and Installed versions A clone might downgrade plugins/apps, and potentially remove plugins/apps and the tables that go with them.

A clone target instance that is on a later version of CC apps/plugins, and has ACCs installed on servers at those later versions, could be overwritten by a source instance on a previous ACC version. Future version installs will now be linked with an instance that is on an earlier version, and code and checks may be incompatible.

A clone source instance that doesn't even have ACC apps installed, won't have the ACC code or tables, and so nor will the clone target instance after the clone. Those tables, the records they contain, and the code to go with them is deleted as part of the clone. The MID Server web server extension context and websocket lister code will no longer exist, and so although the ACC installs will continue to try to communicate with their MID Server, the MID Server will not do anything with those requests. Those orphaned ACCs may need re-installing once ACC is set up in the instance again after the clone.

Solution:

Install and upgrade ACC related plugins on the source instance first, to match what is on the target instance, before cloning. ACC does not need to be set up or in use on the clone source instance, just present.

ACC CIs and records 2 main tables record the ACC details:

Agent Client Collectors [sn_agent_cmdb_ci_agent] The main record. This is a CMDB CI Class. If you are referencing an ACC from another record, this is the one you reference. Agent Client Collector Info [sn_agent_ci_extended_info] Additional fields. Each of the above records has one of these too, and references it. Most of the status columns seen on a list of Agents are actually dot-walked from this record. The default clone settings should make sure records on the target instance are preserved, and source instance ACC records are excluded from the copy during the clone.

There are potential problem though with the CI records in the CMDB, and they are all down to the Table-Per-Partition (TPP) extension model of the CMDB.

sys_class_path When any app or plugin adds tables/classes to the CMDB extended table, a sys_class_path value is randomly created for that table, and stored in the sys_db_object record. When plugins like this are installed independently on different instances, the sys_class_path values will be different.

That means records in the clone target instance, that are preserved, will be set with what is now an invalid sys_class_path value after the clone. You will still see those records in a top level listing of the cmdb, but will not be able to open them, with 'Record not found' errors.

The related agent_extended_info record of the original CI records also gets deleted. (TBC exactly how)

The result is that that new duplicate CI records get automatically created when the pre-existing ACC installs start communicating with the instance after the clone.

Solution:

Install ACC-F on the clone source instance first. After the clone, the target instance will have the same sys_class_path value for the sn_agent_cmdb_ci_agent class, so when ACCs are installed for the target instance, they will not have problems in future clones.

See the Known Error article for more details: TBC

Associated records API Keys There might be a problem with table mid_webserver_api_key_credentials, added in San Diego release, which should probably have an exclude/preserver. TBC

Solution:

Backing up the API Keys, especially before a clone, would be a good idea. They can then be re-imported, or added again afterwards.

ACC Plugins (Sensu Assets) ACC Plugins (aka Sensu Assets) are 'code' and so should be copied in a clone. All records in sn_agent_asset, and their .tar.gz file attachments should be copied over in the clone, replacing any records that were on the target.

However, until the clone automation is fixed, these are being treated as large attachments, and deleted.

See the Known Error article for more details: KB1002379 / PRB1550798 Clones that exclude large attachments break Agent Client Collector, by deleting the tar/gz files attached to ACC Plugin records in sn_agent_asset

Solution:

Repair all the Agent Client Collector plugins to get the attachments back.

Extension Context records These extensions need to be preserved/excluded in clones:

MID Web Server ACC Websocket Endpoint plus any others related to Event Management/Operational Intelligence etc. The instance clone automation hasn't been very good at excluding or preserving individual Classes within a TPC extended table hierarchy. This results in corrupted records that can't be opened and run properly by the mid server, and then duplicate records getting created that might appear to be the only record.

You can get an overview of all extension in the ecc_agent_ext_context table, because everything is a child table of that.

Some record may not open with the error "Record Not Found".

Known problems with the out-of-box preservers/excludes:

PRB1408738 Agent Client Collector tables are missing Clone Exclude/Preserver settings - Fixed since ACC- F 2.1.41. KB1212634 / PRB1628241 Clone Excludes/Preservers are missing for SNMP Trap and vCenter Event based Discovery MID Server extension contexts (ecc_agent_ext_context_trap / ecc_agent_ext_context_vcenter) KB1212633 / PRB1628236 Clone Excludes/Preservers are missing for Metric Intelligence MID Server extension contexts (ecc_agent_ext_context_metric) KB1212632 / PRB1628223 Clone Excludes/Preservers are missing for Event Management MID Server extension contexts (ecc_agent_ext_context_event / eif_listener_context) These are all fixed in Vancouver, however there may still be Clone engine known problems that cause this when not every table in the extended table hierarchy is exclude and preserved in the clone.

Solution:

To repair bad records, please use this KB article: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0716609

To find the records, set the script up with: findOrphans('ecc_agent_ext_context', null, false);

Then to delete them use: findOrphans('ecc_agent_ext_context', null, false);

You may then need to set up the MID Server extensions/collectors/listeners again.

Prevention:

To prevent this in future, you will need to add Clone excludes and preservers for ecc_agent_ext_context and all the tables that extend it.

You can see a list of what child tables you have in your instance using this list filter: https:// .service-now.com/sys_db_object_list.do?sysparm_query=super_class.label%3DMID%20Server%20Extension%20Context%5EORsuper_class.super_class.label%3DMID%20Server%20Extension%20Context&sysparm_first_row=1&sysparm_view=