Agent Client Collector Upgrade TroubleshootingSummary<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Check the ACC-F Release Notes for more recent changes. Table of Contents IntroductionGlossaryDocumentation LinksOS and App Version support matrixRequired permissions LinuxWindows Downloading the installers Downloading from the Install serverDownloading from an alternate URLOn-Premise/Self-Hosted customers Overview of stepsDetailed Steps, including what Code and Known Problems are involved at each step User starts the processInstance VerificationAgent Verification - triggers CheckAgent Verification - In the AgentAgent Verification - Sensor in InstanceUpgrade - The scheduled upgrade process runsUpgrade - Get the LogUpgrade - Get the Log - in the ACCUpgrade - Get the Log - Sensor Failed because of 30 minute Timeout Introduction ACC-F 2.9 introduced a Selective Upgrade feature to enable self-upgrade of existing ACC installs triggered from the ServiceNow Instance, and this KB is aimed at helping Customers and Tech Support understand and troubleshoot that process. Note: This feature is not designed to replace 3rd party deployment/automation tools (Chef, Puppet, Ansible, SCCM, etc.), but to help fill in the gaps in Cloud or DMZ networks where the normal automation cannot reach. This feature is not designed to do mass upgrades of thousands of installs, and should not be expected to be able to do that. This KB does not cover upgrades by 3rd party deployment/automation tools. Glossary ACC = Agent Client Collector = Agent = an install of ACC on a server/PCACC-F = Agent Client Collector Framework app, contains all the Upgrade logicAsset = ACC Plugin = Sensu Asset - https://docs.sensu.io/sensu-go/latest/plugins/assets/App = Store App = like an "Instance Plugin" but out-of-bandKB0957773 Agent Client Collector Acronyms and Abbreviations explained Documentation Links Linux Upgrade an agent in an instance - Includes specific commands run by the upgrade. OS and App Version support matrix OS*WindowsLinuxMacOSVersion adding the upgrade code2.92.9Not supported yetEarliest version that can be upgraded2.72.7Not supported yetNotes * See the store release notes for the versions that add specific OS edition/version/distribution support. This works on instance versions starting from Paris, although by the time this was released instances should have upgraded past that. The MID Server is tied to the instance version, and does have ACC code compiled in, but as this upgrade process is implemented as any other ACC Check there are no additional dependencies in the MID Server code on top of what is required present to run any Checks. Required permissions Linux Enable sudo for the ACC user for the following commands: systemctl start accsystemctl stop accsystemctl daemon-reloadFor RedHat/CentOS:/usr/bin/rpm –Uv <cache folder>/upgrade/agent-client-collector-upgrade.rpmFor Debian/Ubuntu:/usr/bin/dpkg --install --refuse-downgrade --skip-same-version <cache folder>/upgrade/agent-client-collector-upgrade.deb Windows The ACC Service needs to be running as a user that is a member of the local Administrators group. Note: This level of privileges is higher than you would need in general for an Agent to run. You may want to continue using your automation for upgrades, and keep the permissions as low as needed. For example, Upgrades requires Admin, but below is all you need for ACC for Visibility Discovery, and only admin(SYSTEM) is needed for a couple of things you may not need to discover at all. Downloading the installers ACC downloads directly from the install server by default. ACC installs will need to access this URL:https://install.service-now.com/ Network connectivity requirements for ACC Upgrades are the same as for MID Servers, and similar install server firewall rules will need to be in place for ACC hosts as for MID Servers and so the MID Server docs can be used for the current information:Docs: Download the MID Server files (Washington)"The IP address of the MID Server download site (install.service-now.com) can change without notice. To ensure that you can download the MID Server installation package and receive automatic MID Server upgrades, allow local network access to these IP addresses: 149.96.5.98149.96.6.98 " - retrieved 23/5/24 If you do a DNS lookup on install.service-now.com, and you don't get one of those, then both will have changed, and the docs will need updating. As a customer, you have no way of telling what the 'other' IP is. If you get a browser error when clicking that on the ACC host server, or MID Server host, then you are blocking access to that URL somewhere, and will need to get internal IT network/security involved. If you get a blank page back, but with no errors ,you are good to go. Downloading from the Install server Example error if the install server is blocked by a firewall (e.g. a 2.7.0 to 2.10.1 upgrade for Windows): Agent upgrade Failed. Check output: Something went wrong while trying to download https://install.service-now.com/glide/distribution/builds/package/app-signed/agent-client-collector-2.10.1-windows-x64.msi and save it to C:\Windows\Temp\ACC_Upgrade\agent-client-collector-upgrade.msi: Failed to open TCP connection to install.service-now.com:443 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect(2) for "install.service-now.com" port 443). ["C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/net/http.rb:960:in `initialize'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/net/http.rb:960:in `open'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/net/http.rb:960:in `block in connect'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/timeout.rb:105:in `timeout'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/net/http.rb:958:in `connect'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/net/http.rb:943:in `do_start'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/net/http.rb:932:in `start'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:346:in `open_http'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:764:in `buffer_open'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:235:in `block in open_loop'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:233:in `catch'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:233:in `open_loop'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:174:in `open_uri'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:744:in `open'", "C:/Program Files/ServiceNow/agent-client-collector/embedded/lib/ruby/2.7.0/open-uri.rb:50:in `open'", "C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-commons/bin/common_utils.rb:378:in `downloadFile'", "C:/ProgramData/ServiceNow/agent-client-collector/cache/acc-f-upgrade-msi/bin/runUpgrade.rb:189:in `<main>'"] Downloading from an alternate URL This can be done by adding parameter agent-upgrade-url-path to the acc.yml file, with the URL of the web server folder that has the installer files. You would first manually download all the different OS installers from the links in the instance, and copy them to the web server. The usual rules about trusted web server SSL/TlS certificates will apply. Using a self-signed, or internal CA generated certificate for the web server may cause problems. e.g. acc.yml agent-upgrade-url-path: https://127.0.0.1:8800/static/acc_installers/ On-Premise/Self-Hosted customers If you are on ACC v3.0.0 or later, and you've never had any problems with MID Server auto-upgrades failing, then ACCs should also have no problem downloading the installer file. If you cannot provide access to the install server, or an alternate web server URL, then it may mean you have to use your existing provisioning automation to also do the upgrades. Overview of steps Instance Verification Instance-side tests: Up, OS Type, Versions Agent Verification Agent side tests: Runs a Check in the ACC: Can Download, Permissions Upgrade Runs the upgrader, Gets the final log Detailed Steps, including what Code and Known Problems are involved at each step User starts the process The upgrade process is started by either of these UI Actions, which required the agent state to be "Up" to be displayed. The "Upgrade Agent" related link on a /sn_agent_cmdb_ci_agent.do formThe "Upgrade Agent" list action on a /sn_agent_cmdb_ci_agent_list.do, after checking some rows. If you include some in your selection that are Down, only the Up ones are passed through to the UI action. To see these UI actions, the user must have the agent_client_collector_admin role.A Confirmation popup is displayed This is UI page "upgradeAgentConfirmation", that contains both the HTML and Client Script for the popup."More than 20 agents had been selected. Please select fewer agents for upgrade and try again" is displayed if you did, and this limit is hard coded in that UI Page. This is not intended to be altered by the customer.KB1157949 Can all Agent Client Collectors (ACC) be upgraded at the same time?If 20 or fewer Agents were selected, the "Upgrade" button is shown.Potential problem: If an Up and Down Agents is selected, you can still click "Upgrade Agent (1 of 2)", and both sys_ids are passed to the ajax request, and the confirmation popup will say "Are you sure you want to upgrade 2 agents". Only 1 actually upgrades and gets an upgrade history record, so this is probably cosmetic. Click "Upgrade" The UI Page Client Script makes an ajax request to Script Include "AgentUpgradeAjax", UpgradeAgents function, with list of agent sys_ids.This can be seen in instance app node logs as a /xmlhttp.do user session transaction, where sysparm_agentSysIdList are the sn_agent_cmdb_ci_agent sys_ids. If there is an exception/crash, you will see it as part of this transaction in the app node log. e.g. 2022-08-23 03:48:44 (087) Default-thread-2 4AD49B2097A9115409D530FFE153AFC2 txid=f77c9b64976d #2434813 /xmlhttp.do Parameters ------------------------- sysparm_isList=false sysparm_agentSysIdList=8f2541051bc7c150c27d5465604bcb31 sysparm_scope=sn_agent sysparm_processor=AgentUpgradeAjax sysparm_want_session_messages=true sysparm_name=upgradeAgents 2022-08-25 07:17:37 (696) Default-thread-15 1E0C5E994725D59C1CEFFDA8536D4336 txid=cb7f5e5d4765 #6452025 /xmlhttp.do Parameters ------------------------- sysparm_isList=true sysparm_agentSysIdList=685f9b501b01011044846421b24bcbc1,8f2541051bc7c150c27d5465604bcb31 sysparm_scope=sn_agent sysparm_processor=AgentUpgradeAjax sysparm_want_session_messages=true sysparm_name=upgradeAgents For each selected agent, a sn_agent_upgrade_history record is created for Instance Verification, with State=In Progress. The Status, Messages and additional records are created in this table as the upgrade progresses. The agent_client_collector_admin role is required to see these records.This is a synchronous Ajax request, so the request has to complete without crashing, and then you are redirected back to: The Agent form, for single upgrades You may get errors from Instance Verification displayed immediately at the top of the page. The sn_agent_upgrade_history listing, filtered for only the records created at or after the upgrade time, for any agent. The listing might include log entries for other upgrades that were already in progress, or triggered after.Known Problem: (in 2.10.1): PRB1603893 / KB1169916 ACC Upgrade redirects to a list of Upgrade History records, and the filter timestamp is wrong, so includes previous upgrades as well The sn_agent_upgrade_history record "Unique Sequence per upgrade" value (upgradeGuid) is per upgrade, per agent, allowing all logs for a specific Agent's upgrade to be grouped manually too. Instance Verification AgentUpgradeAjax UpgradeAgents() continues, and will run upgradeSingleAgent() for each agent sys_id submitted.All but the last 2 upgrades for the agent are deleted from sn_agent_upgrade_history, by script include AgentUpgradeUtil, function prepareTable. System property sn_agent.min_upgrade_history_per_agent can be used to change that number.The 'InstanceVerification' record will be updated with state 'Failed', and a message included for: "Could not locate agent based on sysid=" if the sys_id is not found in sn_agent_cmdb_ci_agent."Could not locate agent based on agent_id=" if the sn_agent_cmdb_ci_agent record is found, but the sn_agent_ci_extended_info record is missing."Could not deterine agent version for agent with sysid=xx. We got: " [sic] if the sn_agent_ci_extended_info does not have a version value for the currently installed version on the host."Could not locate Computer with sysid=" if the sn_agent_cmdb_ci_agent record "cmdb_ci" field reference is not found in cmdb_ci_computer (or extending tables).Known Problem: (in 2.10.1): PRB1602489 / KB1167250 ACC Upgrade Instance Verification stuck on In Progress state, caused by NullPointerException because of no linked CI on Agent "The agent OS version (xxx) is currently not supported." is worked out by finding the "os" value as an entry in the JSON object "upgradeRequirements", in the AgentUpgradeAjax script include. That specified which check definition to use. If you get an error for an OS/Linux distribution the release notes suggest is supported, then the AgentUpgradeAjax script include might not be the latest version, or some values were missed out. The following is the v3.5.3 list.: Debian/Ubuntu: GNU/Linux 9GNU/Linux 10GNU/Linux 11Linux Debian 12 (from v3.5.3/PRB1727743)Ubuntu 18Ubuntu 20Ubuntu 21Ubuntu 22 (from v3.4.0/PRB1633300) RedHat/CentOS/SuSE/Oracle/Alma: Linux Red Hat 7Linux Red Hat 8Linux CentOS 7Linux CentOS 8Linux SuSE 12Linux SuSE 15Linux Alma 9 (from v3.4.0/PRB1691307)Linux Oracle 9 (from v3.4.0/PRB1633300)Potential problem: 2.10.2: "Supporting new OS Versions... CentOS Stream 8 and 9, RHEL 9" , 2.5.0: "New:OS Support – Amazon Linux 2 and Oracle Linux". not in that list. Confirmed: Some OS versions that the Installer has been tested with have not yet been tested for the Selective Upgrade.Potential problem: Oracle Linux might be being recorded in the CI os field as Red Hat (PRB1569423 related?)Potential problem: Discovery Linux pattern does not name the os/version this way, which will cause flapping or failures if the CI has the Discovery value at the time. Note: The fix for PRB1633300 in v3.4.0 changed the script include so it is no longer Linux OS version specific. This should avoid issues with newer OS versions, without needing code updates each time something is released.If the linked host computer CI's "os" value contains "Windows" in it. e.g. "Windows Server 2019", then the OS we check for is always "Windows". Otherwise the value to check for is: os + " " + version. e.g. "Linux Red Hat 7".Potential problem: The "Windows" check won't realise if an Agent is installed on a new version/edition of windows not officially supported/tested yet.Potential problem: Other CMDB data sources (Discovery/Service Graph Connector/...) may not be using the same os/version naming convention as ACC expects, and that will cause the OS check to fail. IRE Reconciliation rules may be needed to keep the value as this code expects. "The agent's current version is not the minimum agent version for upgrade. Minimum version for {x} is {y}". This is also from the same JSON in AgentUpgradeAjax script include, using the MinAgentVersion value. For Windows/Linux that was initially version 2.7 minimum."Agent xxx is not currently up, canceling upgrade." if the Agent went Down since clicking Upgrade Agent earlier. The 'InstanceVerification' record will be updated with state 'Skipped', and a message included for: "Agent version is equal to (or higher than) ACC-F version; nothing to upgrade." is checked by script include AgentVersionFieldStyle, function isAgentVersionLowerThanAccfVersion.Known Problem: (in 2.10.1) PRB1590846 / KB1157996 ACC Upgrade: Agent version is equal to (or higher than) ACC-F version; nothing to upgrade - When attempting a valid minor version upgrade e.g. 2.9.0 to 2.9.2 / 2.10.0 to 2.10.1KB1156997 Can the "Upgrade Agent" feature be used to Downgrade an Agent Client Collector (ACC) If all of those checks were OK, the 'InstanceVerification' record will be updated with state 'Success', and a message: "Instance validation success. Invoking upgrade on agent." Agent Verification - triggers Check AgentUpgradeAjax UpgradeAgents() continues, and will run a different check, depending on the OS, as specified in the upgradeRequirements JSON in AgentUpgradeAjax script include: For Debian/Ubuntu: "Agent Upgrade - DEB", using plugin "acc-f-upgrade-deb".For RedHat/CentOS: "Agent Upgrade - RPM", using plugin "acc-f-upgrade-rpm"For Windows: "Agent Upgrade - MSI", using plugin "acc-f-upgrade-msi" This does not check if the Windows version/edition is a supported one. If an ACC got installed somehow on a very new version, it's going to try and upgrade anyway, which may not work. The Agent Verification, and Upgrade are executed as a check using scripts in these plugins. Known problems in previous version will still be seen if the assets are still the old version.Known Problem (2.10.1 fix): PRB1554342 / KB1158004 Opening/Downloading an ACC Plugin (asset) record "touches" it, preventing it from being updated during next store app upgradeKB1157033 How to revert ACC Plugins (assets) to out-of-box versions to solve Skipped upgrade related code mismatch issuesCheck Definitions of type=Upgrade exist for these checks, but there are no Check Instances or Policies set to use them. Instead they are launched directly by the Upgrade code in the instance.The target version number is appended onto the command for the check e.g. "runUpgrade.rb -v 2.10.1". This is done by the injectCommandSuffixBasedOnCheckType() function of the AgentNowHandler script include.Known problem (in 2.10.1): PRB1598028 / Agent Client Collector Selective Upgrade does not upgrade to latest minor version - rounds down to x.y.0 (2.9.0 vs. 2.9.2 / 2.10.0 vs. 2.10.1)The check is launched by AgentNowHandler script include, runCheckForCis function, with a 10 minute timeout, at Expedited priority. The upgrade check should jump the ECC queue, if it has a backlog of Standard priority checks queued up, and be executed sooner by the MID Server.The ecc_queue output for the check. (Windows example)A new sn_agent_upgrade_history record for upgrade stage 'AgentVerification' will be inserted, with state 'In Progress', and message: "Starting agent side validations..." The AgentUpgradeAjax UpgradeAgents function is now finished. The /xmlhttp.do user transaction will end. Everything from now on happens in the background. Agent Verification - In the Agent The MID Server has to be able to pick up that job and process it, before the ACC will run the check and return the result in an ecc_queue input.The Agent runs the check command, which is Ruby file runUpgrade.rb, from the plugin/asset of the OS specific check definition, which should already have been synched to the Agent's asset/plugin cache, via the agent\static folder tree of the MID Server Web Server Extension, which was synched from the instance Plugin attachment to the MID Server.KB0852276 How MID Server File Synchronisation works - Troubleshooting guideThe asset file will need the certificate verifying if Certificate Verification is enabled in the acc.yml file.Known Problem: PRB1590658 / KB1158506 Windows ACC Upgrade fails during ACC Plugin/asset signature verificationThe script logs to %TEMP%\ACC_Logs\ACC_Upgrade.logThe script knows the version it needs to upgrade to from the command line parameter. e.g. "runUpgrade.rb -v 2.10.1". If this is missing "Version is mandatory in order to operate. Please supply the version for the upgrade." is logged, or if it doesn't look like an x.y.z version "Version xxx does not meet the version definition major.minor.sub".Windows: The Agent Client Collector (AgentClientCollector) windows service logon as user is checked, to make sure it has the Admin requirement. If the check fails, logs "Windows upgrade must be performed by an administrator". When successful, depending on the type of admin user, logs: SYSTEM: "INFO -- : Checking if nt authority\system is an admin?" "INFO -- : SYSTEM account is admin!"Local Admin: "INFO -- : Local admin!"Domain Admin: "INFO -- : Domain admin!"Known Problem: PRB1772701 / KB1649666 Upgrade Agent fails with error "Windows upgrade must be performed by an administrator" if the Administrators group name is not in English, or is given administrator rights via Group Policy "Service AgentClientCollector is NOT running" if the Windows Service is not running. The service name is hard code to the default "AgentClientCollector" in the script.Checks the version currently is lower than the version to upgrade to, by running "acc.exe version" to find the current version. If the records in the instance have accurate version numbers, this should have been prevented in Instance Verification. "Could not determine installed version." if that command didn't run or the output couldn't be parsed. "Installed version xxx is equal to the upgrade request (yyy) and is already installed""Installed version xxx, which is larger from the upgrade request (yyy), is already installed"Known Problem: PRB1585421 / KB1157970 Windows ACC-F upgrade error: "Installed version 2.9.0, which is larger from the upgrade request (2.10.0), is already installed", when it isn't (fixed in 2.10.1) The upgrader file, for the specific upgrade version, is downloaded from the install server https//install.service-now.com/ and so firewall rules for the URL or the IP pair (149.96.5.98/149.96.6.98 at time of writing) need to be in place for all Agent hosts computers. Logs:"INFO -- : About to download MSI file from https://install.service-now.com/glide/distribution/builds/package/app-signed/agent-client-collector-2.10.1-windows-x64.msi"If the download aborts for some reason, the error will be logged, otherwise logs "INFO -- : Download completed"Checks the file is actually there, and if not logs "Could not find upgrade file xxx Something went wrong. Exit."Execution of a script to execute the upgrader is scheduled. This then continues in the background, and continues to log to ACC_Upgrade.log file.*** template xml file and steps NEEDS EXPANDING ON *** Linux - Debian check output can contain the failure to run specific sudo command on Linux. See section on Required Permissions.*** TODO - these runUpgrade.rb scripts are considerably different per Asset *** Linux - Redhat check output can contain the failure to run specific sudo command on Linux. See section on Required Permissions.*** TODO - these runUpgrade.rb scripts are considerably different per Asset ***The runUpgrade.rb script ends The Check result, which includes the current contents of the ACC_Upgrade.log file, is returned to the instance in an ecc_queue input. Agent Verification - Sensor in Instance The ecc_queue input payload will look something like:The ecc_queue input will be processed by the sensor as a "Upgrade" type check. The AgentNowHandler script include processEccRecord() uses table sn_agent_check_type to run AgentUpgradeUtil script include processAgentUpgradeValidationResponse function.If there has been any tests failed or problem with the Check execution, errors may be: "Did not get any check results. Upgrade state unknown." if nothing was passed to the processAgentUpgradeValidationResponse function"Could not find sequence ID in check results. Check = {check}"Could not locate agnet with agent id {agentId}" [sic] - no sn_agent_cmdb_ci_agent record found with the agent_id from the payload"Could not locate agent id {agentId}. Upgrade state unknown." - no sn_agent_ci_extended_inforecord found with the agent_id from the payload, so we don't know the State."Did not get enough details. Cannot continue with upgrade." if any of the above fails. The 'InstanceVerification' record will be updated: state 'Failed', and a message containing the full output, if the first line of the output does NOT contain the word "success". Upgrade goes no further.state 'Skipped', and a message containing the full output, if "is already installed" appears anywhere in the output. Upgrade goes no further.state 'Success', and a message starting 'Agent upgrade validation successful. Upgrade will now start and may take up to 30 minutes depand on the OS.'[sic] followed by the full output. e.g. If it's a Success, a new sn_agent_upgrade_history record for upgrade stage 'Upgrade' will be inserted, with state 'In Progress', and message: "Agent upgrade is being performed." The agent record sn_agent_ci_extended_info is set to state "Upgrading". Upgrade - The scheduled upgrade process runs The Scheduler Task started by the Check performs all the steps of the upgrade template XML file, (stops the service, runs the installer, starts it again) and eventually the Agent will start up again on the new VersionThe installer will leave a log file here, recording what it did and any errors: Windows: %TEMP%\ACC_Logs\MSI_ACC_Upgrade.log If running as SYSTEM, that's C:\Windows\Temp\ACC_Logs\MSI_ACC_Upgrade.logOtherwise C:\Users\<service username>\AppData\Local\Temp\ACC_Logs\MSI_ACC_Upgrade.log Linux: /var/log/servicenow/agent-client-collector/upgrade.log The agent record in the instance get set to State "Up", and the version field updated, assuming the Agent was able to be started! If it stays down, you will need to manually get those log files from the host. Upgrade - Get the Log When the Agent comes back up, and checks back in with the instance, the MonitoringConfig script include classifyClientsAndUpdate() method will run, as it would even if there was no upgrade.That runs AgentUpgradeUtil script include agentUpgradeToUp() function, which does this: "Could not locate agent record for agent id {agentId}" if the sn_agent_cmdb_ci_agent record is not found.The sn_agent_upgrade_history record for upgrade stage 'Upgrade' will be updated, with state 'Success', and message: "Fetching upgrade log." invokes the "Fetch upgrade log file" Check Definition to fetch the upgrade log file, Expedited priority, with a 10 minute timeout. This does not use a plugin/asset, instead uses existing "read-file.rb", which is the same ruby script used by things like "Grab Agent Config".The only parameters passed are agnetHistorySysId, so that the result can be linked back to the correct "Upgrade" stage sn_agent_upgrade_history record, and file_path, which will depending on whether it is Windows or not, and is based on agent/environment variables so should work even if the install location is not default.ecc_queue output looks like: Upgrade - Get the Log - in the ACC TODO - Show acc.log as check runs. Upgrade - Get the Log - Sensor The ecc_queue input payload will look something like:The ecc_queue input will be processed by the sensor as a "Upgrade log file" type check. The AgentNowHandler script include processEccRecord() uses table sn_agent_check_type to run AgentUpgradeUtil script include processFetchedUpgradeLog() function.Errors might include: "processFetchedUpgradeLog: Did not get any check results." if there was no output from the check"processFetchedUpgradeLog: no history sys id." is the history record sys_id is missing"processFetchedUpgradeLog: Did not locate history sys id." is there is a history parameter, but it cannot be found in sn_agent_upgrade_history If the text "upgrade success" is found in the last couple of lines of the log, State is "Success", otherwise "Failed". 'is already installed'The sn_agent_upgrade_history record for upgrade stage 'Upgrade' will be updated with the log in the message field, and State as: if the text "upgrade success" is found in the last couple of lines of the log, State is "Success"if the text "is already installed" is found in the log, State is "Skipped"else, State is "Failed" Failed because of 30 minute Timeout If any of the above steps gets stuck, crashes, never completes, etc. there is a overall timeout for the upgrade process controlled by AgentKeepAlive.If an Agent spends more than 30 minutes (property sn_agent.agent_upgrade_wait_time) in Upgrading state, then the AgentKeepAlive script include will: call markFailedIfTimeout() from the AgentUpgradeUtil script include, which will update all "AgentVerification" and "Upgrade" sn_agent_upgrade_history records that were "In Progress" state, to "Failed" state with message "Timeout reached".Change the Agent record status to Down Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Written for the initial ACC-F v2.9 release. Note: Self-upgrade has been expanded to upgrade larger numbers of batches of agents since this was written.