Manage outstanding issues with MID Servers

Product Success Playbook

A step-by-step guide to analyze and remediate issues with MID Servers

Table of Contents

Summary

Goal of this Playbook

Audience

Problem Overview

Executive Summary

How this playbook can help you achieve business goals

How this playbook is structured

Problem Analysis

Upstream Causes

Downstream Consequences

Impact on Your Business

Engagement Questions

Remediation Plays

Summary

Play 1: Review your data

Play 2: Fix Play

Play 3: Fix Play

Data Governance

References

Summary

Goal of this Playbook

This playbook helps in analyzing and resolving outstanding issues with MID Servers

Details about this playbook

Author

Bibu Elias Punnachalil

Reviewer

John (Johnny) Walker

Date

08/04/2022

Addresses HSD #

HSD0010809

Applicable ServiceNow Releases

All Releases

Prerequisites

Time Required

Approximately 4 to 16 hours (depending on your environment)

Audience

Discovery Administrator ServiceNow Administrator

Problem Overview

Issues with MID Server, due to various causes, may result in failure for Discovery , Orchestration, Service Mapping, Service Analytics, Event Management & Cloud Management. It may simply impede performance. Degradation & failure can result in duplicate CIs created from discovery or missing CI records as well as frustration and wasted time.

Executive Summary

How this playbook can help you achieve business goals

This playbook helps in improving the MID Server health which in turn results in a more accurate and complete CMDB and promotes visibility and management of IT Assets

How this playbook is structured

This Playbook will guide you through 4 plays.

Play 1 (a review data play) helps you review MID Server issues Play 2 (remediation) provides remediation steps required to fix errors with short description containing “Unexpected Error” Play 3 (remediation) provides remediation steps required to fix other issues identified and reviewed in Play 1 Play 4 (a Data Governance play) lists the guidelines and processes for continuing to ensure your MID Server issues are minimal

Problem Analysis

Upstream Causes

Invalid user credentials after instance cloning. Unsupported Java Runtime Environment (JRE) on the MID Server host. Approaching resource thresholds for CPU usage and JVM memory. Errors detected during pre-upgrade testing. User authentication and authorization failures. Network connectivity interruptions and role configuration errors MID server tries to decrypt files that are trying to be synced from the instance before it had a chance to be validated Information messages related to license non availability for plugins. Downstream Consequences

Data Consequence

CMDB may have duplicate CI records Missing CIs in CMDB Missing Events Operation Consequence

Operational teams may require spending additional effort in troubleshooting MID Server issues Automated remediations may not get executed ITOM Health integrations might not receive critical infrastructure events Trust in CMDB may be lost due to duplicate and missing CIs

App Consequence

SecOps applications can fail to identify risks in a timely manner ITOM Visibility (Service Mapping) could fail depending on available MID Servers for IP Ranges ITOM Health integrations relying upon data collectors could be failing entirely IntegrationHub activities and flows could be fail entirely for lack of a backup Web Services integrations that utilize a MID Server could be impacted by Down MID Servers

Impact on Your Business

MID Servers issues could impact your business outcomes depending on which ServiceNow applications these MID Servers support. Some examples follow.

Increased MTTR

ITOM Health depends on MID Servers for event data ingestion

CMDB Trust

ITOM Visibility depends on MID Servers to perform Discovery Schedules

Process Automation

Data Sources / Import Sets could be relying on your MID Servers IntegrationHub / ETL transforms could be reliant upon MID Servers

Engagement Questions:

Consider the answers to these questions:

Who manages your MID Servers & do they periodically review issues associated with each Mid Server? Is there a plan in place to handle issues with MID Servers? Is the purpose of each of your MID Servers well understood?

Remediation Plays

Summary

The table below lists and summarizes each of the remediation plays in the playbook. Details are included later.

Play Name

Review your data

What this play is about

Review MID Server Issues

Required tasks

List the active MID Server Issues

Fix Play : Unexpected Error

What this play is about

Errors with short description

Required tasks

Analyze the error and in case not possible

Fix Play 2

What this play is about

This play helps you to fix other MID Server issues

Required tasks

Execute the steps provided

Data Governance

What this play is about

Helps you establish standard operating procedures, as well as periodic data certification

Required tasks

Provides detailed advice, best practices and data audit instructions

Play 1 - Review your data

What this Play is about

Review active MID Server issues

Required tasks

Navigate to MID Server  Server Issues. This will list all active MID Server issues present in the MID Server Issues table. (which is ecc_agent_issue) Compare the short description for the matching the error description in Fix Plays from this playbook

Play 2 – Fix Play : Error short description contains “Unexpected Error”

What this Play is about

Unexpected errors may occur due to various reasons. Handling these issues may become complex

Required tasks

Check if the Unexpected error is due to bundle file extension missing. In this case below support knowledge article https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0855846

While trying to Import an application from Source Control or Link an existing application to Source Control users may face the following error https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0827801

In case error root cause is not identified, the MID Server Service may be reinstalled https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0715632

As a last resort you may reinstall the MID Server. Please refer to the below article for the best practices on setting up a MID Server https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0535145

Play 3 - Fix Play: MID Server

What this Play is about

This fix play provides the support knowledge article reference to the common MID Server issues

Required tasks

Check if MID Server Issue short description matches with any of the below, refer to the associated Knowledge Article in ServiceNow support site for resolution

Error Short Description

Knowledge Article Link from Support

"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" version

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0749654

Mean CPU used %' has exceeded threshold

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0597639

A MID Server with a duplicate name or sys_id was prevented from connecting

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0726915

Aborting MID Server upgrade due to pre-upgrade check failure: Unable to download updates from install server

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0565184

Could not decrypt file discovery whitelist after sync

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0727848

Detected JRE switched from version

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0750004

Error encountered when invoking PowerShell, the result from running

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0752093

MID Server encryption keys do not match and are no longer valid.

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0866897

MID Server failed to upgrade

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0713557

The current service named snc_mid_mid_us_prod does not match the expected service name

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0861098

The home directory changed

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0743043

Unable to load key store, MID server will not able to function (1)

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0959666

Unable to subscribe to AMB channel

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0779124

User ****** associated with 1 down MID Server. No login attempts

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0597574

Data Governance

What this Play is about

This play provides an approach for monitoring and managing the MID Server issues

Required tasks

Rerun Play 1 periodically to check for any new errors, assign this task to a user, for example your Discovery Admin Use the fix plays provided in this playbook to fix the errors Keep monitoring MID Server logs (Agent logs, Wrapper logs). These logs can be pulled from instance using “ Grab MID Logs ” link on MID Server record

Use MID Server pre-upgrade check to automatically test the MID Server's ability to upgrade on the system prior to the actual upgrade, to identify issues that could cause a MID Server outage or require reinstallation

Congratulations

You have completed this Product Success Playbook .