MID Servers should not remain in Down status - Product Knowledge

MID Servers should not remain in Down status  Product Success Playbook

MID Servers should not remain in Down state

A step-by-step guide to analyze and remediate ITOM Health & Visibility issue

Table of Contents

Summary

Goal of this Playbook

Audience

Problem Overview

Executive Summary

How this playbook can help you achieve business goals

How this playbook is structured

Problem Analysis

Upstream Causes

Downstream Consequences

Impact on Your Business

Engagement Questions

Remediation Plays

Summary

Play 1: Review your MID Servers

Play 2: Analyze your MID Server records

Play 3: Fix Play - Remove & Reinstall

Data Governance

Summary

Goal of this Playbook

The goal of this playbook is to help you identify MID Servers that are Down (offline) and remediate any issues with them to return them to the Up (online) state.

Author John (Johnny) Walker Date 11/01/2021 Addresses HSD # HSD0009876 Applicable ServiceNow Releases Any Time Required Approximately 1 to 2 hours (depending on your environment) Audience

ServiceNow Administrator or Discovery Administrator

Windows / Linux Administrator supporting your MID Server installations

Problem Overview

The MID Servers are a necessary part of your ITOM applications on the ServiceNow platform. They perform work within your network and report back to your ServiceNow instance. When these servers are in the Down (offline) state, they are unable to perform the activities assigned to them.

Executive Summary

How this playbook can help you achieve business goals

MID Servers are the gateway to enterprise network and provide visibility into IT infrastructure. Having working MID Servers is essential for the various applications they support. If MID Servers are down, there is potential of losing visibility to all or part of your network infrastructure, which can result in delays, outages and disruptions to business processes.

This playbook will help you to identify the MID Servers you have in the Down status and return them to the fully operational (Up) status.

How this playbook is structured

This playbook will guide you through several plays:

An analysis play designed to help you confirm our findings

Remediation play(s) designed to help you resolve or remediate the findings

Governance is provided to help you keep your system healthy in the future

Problem Analysis

Upstream Causes

Windows patching can lead to java / OS networking stack issues on the host machine

Intrusion protection and Antivirus solutions can interfere with MID server performance, processing, integrity and stability

Firewall, Routing and Proxy changes within your network can interrupt access between the MID server and your SN instance

During upgrades, a MID server could fail to update and restart completely

Decommissioned MID server records (ecc_agent) could remain in your SN instance after having been replaced or retired

The MID Server user could have been deactivated, had the password rotated or somehow lost the mid.server role

During development and testing of new features or patterns, the Java application could hang or crash

An administrator might have stopped the Java process for maintenance or testing and it simply needs to be restarted

Downstream Consequences

Data Consequence

Your CMDB records could be getting stale for lack of Discovery coverage

Your ITOM Health integrations might not receive critical infrastructure events

IntegrationHub data ingestion might not be taking place in a timely fashion Operation Consequence

Discovery Schedules could cancel for lack of resources to complete them on time

IntegrationHub business processes could be failing to occur when expected

Discovery / Platform Administrators might be learning to scroll past / ignore Down MID Servers, leading them to overlook a critically Down record

Outages can be taking longer than they need to due to slower processing of ITOM data App Consequence

SecOps applications can fail to identify risks in a timely manner

Discovery Schedules can take longer than they need to, pushing them outside of the off-peak times they are supposed to run within

ITOM Visibility (Service Mapping) could fail depending on available MID Servers for IP Ranges

ITOM Health integrations relying upon data collectors could be failing entirely

IntegrationHub activities and flows could be fail entirely for lack of a backup

Web Services integrations that utilize a MID Server could be impacted by Down MID Servers

Impact on Your Business

MID Servers being down (offline) could impact your business outcomes depending on which SN applications these MID Servers support. Some examples follow.

Increased MTTR ITOM Health depends on MID Servers for event data ingestion

CMDB Trust ITOM Visibility depends on MID Servers to perform Discovery Schedules

Process Automation Data Sources / Import Sets could be relying on your MID Servers IntegrationHub / ETL transforms could be reliant upon MID Servers

Engagement Questions:

Consider the answers to these questions:

Who manages your MID Servers?

Is there a plan in place to handle issues with MID Servers?

Is the purpose of each of your MID Servers well understood?

Remediation Plays

Summary

The table below lists and summarizes each of the remediation plays in the playbook. Details are included later.

Play Name

Review your data

What this play is about

How to view Down MID Servers Required tasks

Navigate to view MID Servers Analyze Play

What this play is about

Required tasks

Fix Plays

What this play is about

Determine that a MID Server is no longer needed Required tasks

Remove orphaned MID Servers Data Governance

What this play is about

Ensure that when MID Servers are Down for any length of time an investigation is began Required tasks

Create a Flow to open an Incident for Down MID Servers

Play 1 - Review your MID Servers

What this Play is about

The list of MID Servers is readily available within your instances. This play reminds you where that data is.

Required tasks

Navigate to MID Servers > Servers

Fill in the Condition Builder as follows and click Run:

Field Operator Value Status is Down Stopped Relative > before 15 > Minutes > ago

Example:

(Optional) You can also navigate to MID Servers > Dashboard and view this information and more.

Example:

Review the records listed here. Take note of the Last refreshed and Stopped fields. These can indicate how long ago communication was received from your MID Servers.

Continue to the remediation play to work on these records.

Play 2 - Analyze your MID Server records

What this Play is about

This play is some common sense advice on reviewing the records from the review play above.

Notes

When reviewing the records in your MID Server (ecc_agent) table, you may or may not immediately recognize each of these records.

MID Servers are created when they are installed on host machine and configured with a login credential which has been granted the mid_server role.

MID Server status changes to Down when they are restarting - due to upgrade or scheduled maintenance on the host machine, or manually for troubleshooting.

MID Server status can also change to Down when communication is lost between the MID (java) process and your ServiceNow instance for a duration of time exceeding ~5 mins.

In some circumstances the java process can appear to be running but communication to your instance may have stopped. There can be many reasons for this. It happens from time to time.

If the MID Server has crashed (the process is no longer running) or the MID can no longer communicate to the SN instance, you must restart the MID from the host machine.

Required tasks

If you are immediately aware that a particular MID Server (ecc_agent) record is no longer valid, simply select it in the list and delete it. These records serve no purpose if they are Down and if they no longer exist, they will never show Up again.

If you are uncertain about a MID Server, you should either use Remote Desktop or an SSH connection to the host machine to start / restart the MID process on that machine. For assistance with this you might need to contact the support team that has access to your MID hosts. This process differs per customer and is up to you to navigate.

If restarting the MID Server process is successful then you can continue on to starting / restarting any other MID Servers which are in the Down status. Return to the List View in Play 1 to confirm that all MID Servers are showing as Up. If when trying to restart the MID process you encounter difficulties you should review the MID Server Landing page collection of KB articles. Many of the common issues with MID Servers are already well documented there. If none of the articles cover the situation you are experiencing, you can also search the Support knowledge base with the specific errors you might encounter in the logs.

If you still cannot start the MID process you have two main options:

Option

Pros Cons Requirements Open a support case for assistance We are always happy to help Could take a day or so to complete several rounds of troubleshooting. Might require a reinstall anyway. Login to the Support (formerly HI) site Reinstall the MID server Eliminates many issues. Might be required as part of a support ticket. Requires a small amount of system engineering. Not much more than installation of a MID server required in the 1st place. Administrative access to the MID host such as Remote Desktop or SSH, and the willingness to do it! Since opening a support case is somewhat self explanatory, continue to the next section to learn how to perform an in place replacement of the MID Server on windows.

Play 3 - Fix Play - Remove & Reinstall MID Server

What this Play is about

How to replace the MID Server with a fresh downloaded version of the software in the event a MID process refuses to start.

The steps below are also located in the ServiceNow Documentation here .

Required tasks

(optional) To uninstall a manually installed Windows MID Server, navigate to the directory where the agent is located. From the agent/bin directory, run the file UninstallMID-NT.bat - we recommend performing this within a command prompt window to view the output.

(optional) To remove a Windows MID Server installed using the guided installation, navigate to Control Panel > Programs > Programs and Features > Uninstall a program and remove the program matching the MID Server's name.

Once the server is uninstalled you can either perform a manual reinstallation or guided installation using the steps here.

If you have removed and reinstalled the MID Server agent on your host machine but are unable to get the new MID Server to connect to your SN instance, refer to the MID Server Landing page and the section titled MID Server Down & Connectivity. There are quite a few articles here that can help you with most common situations.

If your efforts to reinstall the MID Server and connect to your instance are unsuccessful please open a Support Case for assistance.

Once you have installed a new MID Server using one of the methods above it should safe to Delete the MID Server (ecc_agent) record which is still showing in status Down. It may be a good idea to take note of which MID Clusters your MID was previously a part of, and be sure to configure this new MID to take the place of the previous one. Instructions on MID Server Clusters is located here .

Data Governance

What this Play is about

The best means to ensure that your MID Servers are available to the applications and processes they support is to ensure that the fact a MID Server has gone Down is presented to support persons who can investigate and remediate the situation.

You may already be aware there is a MID Server Down Notification active in your instance at baseline. It may be a good idea to review which Users and Groups this notification is configured to send to.

Required tasks

Navigate to System Notification > Email > Notifications

Search for the MID Server Down Notification and open it

Review the "Who will receive" tab to ensure there is someone or some group configured to receive this notification. It may be a good idea to also to ensure that the notification is Active.

Example:

You may decide that a more substantial record demanding action for your Down MID Server is appropriate. In that case, you may decide to put a Flow in place to generate an Incident when a MID Server is Down and remains down for over 15mins. Since the MID Server can be taken down during upgrade or patching to the host machine, it's usually a good idea to allow a small period of time before opening an incident in relation to a Down MID Server.

The following steps are completely optional based on the specific needs of your organization.

Navigate to Process Automation > Flow Designer

Click "New" and select "Flow" from the list presented

Configure the Flow properties fields as shown:

Example:

You may be presented with a welcome message offering you a tour. You can take the tour or not, but we will continue as if you selected to Skip tour.

Select to "Add a trigger" and complete the form as shown and then select "Done".

Example:

Add an Action of type "Wait For Condition". We want to wait for 15 minutes (optionally longer if your needs are different) so it is important to select "Enable timeout" and set 15 minutes in the Duration field. Select "Done" when the form appears as shown.

Example:

We want to check the MID Server Status again after the timeout to ensure we don't open unnecessary incident records. So click "Add an action", select "Flow Logic" and then choose "If".

Complete the step as shown.

Example:

Click the plus sign next to "then" below this condition to add the create task action.

Complete the Create Task form as shown and click "Done".

Example:

Save and then Activate this Flow.

Test the Flow by stopping a MID Server using the UI Action on the form view. Be sure that the MID Server you are stopping isn't currently being used.

Congratulations You have completed this Product Success Playbook.