Linux Discovery – Troubleshooting guideSummary<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Linux underpins most servers, cloud instances, containers, appliances, and network/security tools. It's where business apps actually run, so accurate inventory and relationships directly affect uptime, compliance, and cost. ServiceNow Discovery benefits customers by automating the collection of identity, hardware, OS, network, storage, and cloud metadata—reducing manual effort and drift. ServiceNow Discovery is a 4-phase process where we have the Shazzam, Classification, Identification, and Exploration phases. This KB focuses exclusively on the Linux pattern and provides phase-by-phase troubleshooting steps. Facts<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Release<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } ALL Instructions<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } 0) TL;DR – 10‑Minute First Response Ask these 4 questions up front: Credentials validated? (Username/Key/Password; sudo NOPASSWD for required cmds)SSH reachable on the right port? (default 22 unless changed)Followed the triage flow below end‑to‑end?Any open PRBs/Store updates? (if symptoms look systemic or OOTB pattern‑related) Quick checks (do in order) MID ↔ Target network: ping <ip>, nc -vz <ip> 22, traceroute <ip> (from MID).DNS working: nslookup <ip> / nslookup <hostname> (from MID).Credential test: Use Test Credentials; if fails, SSH manually from MID and run Appendix A commands.Plugins/content: Patterns app + latest Visibility Content installed.Supported OS & shell: English‑based Linux; /bin/sh/bash available.MID health: Java heap OK, ECC queue normal, probe latency reasonable.Firewall/ACLs: TCP/22 allowed, UDP/53 allowed, ICMP optional. If any quick check fails, fix that first before re‑running discovery. PART I — How probes actually fire (what to verify at each hop) 1) End‑to‑end ECC chain you should see Use Discovery Status → Related Links → ECC Queue (or the ECC breadcrumbs/correlator) to follow this order: Shazzam → MultiProbe (UNIX – Classify) — lists scanners and preliminary shell checks.Classification (via SSHCommand/JavascriptProbe): runs uname -a / vendor scripts; sets use_class to Unix classifier; passes deviceHistoryParams.HorizontalDiscoveryProbe – Pattern Launcher: Linux Server — launches the Linux Server pattern and lists any triggered_probes (additional patterns).Pattern probes (MultiProbe / SSHCommand) — e.g., package munger, process list, lsof, storage, fqdn, etc.HorizontalDiscoverySensor — parses pattern JSON, runs pre/post hooks, then sends payload to IRE.Identification/Exploration (IRE) — either insert or update CI and writes related lists. Tip: In the ECC record, the Response to field links each hop; the Agent shows which MID handled it; the Topic shows the probe/sensor type. PART II — Phase‑by‑phase runbook Valid credentials are the single biggest predictor of success across Classification and the Horizontal Pattern phases. If creds don't work non-interactively, discovery stalls before patterns can run or sends empty payloads to IRE. Verify Credentials in Discovery: Go to Discovery > Credentials > Credentials and check: Ensure valid Linux SSH credentials are present.Run "Test Credential" for a failing server.If the test fails, confirm the user has sufficient sudo privileges (if needed). Fix: Add the correct user and grant sudo permissions if Discovery requires elevated access. In case the credential test fails, it is always suggested to check the MID server Agent logs with the DEBUG log level. Also, make sure to "Enable SSH Debug on a MID server"; this can help in printing the SSH connectivity-related ERRORs to the log file. Please check the KB0727653 for more details. Related Knowledge Articles: Credentials & Permissions troubleshooting on Discovery, Service Mapping, Orchestration: KB0657528 Test Credentials work, but the discovery is failing on UNIX/Linux CI: KB0718077 Linux discovery fails with credential errors when using the correct credentials: KB0821304The most COMMON ERRORS that we can see: ✦༻❁༺✦✦༻❁༺✦✦༻❁༺✦ J2SSH [CLIENT] ✦༻❁༺✦✦༻❁༺✦✦༻❁༺✦ Line 4632: 2025-09-04 18:59:07-0400 INFO (Transport protocol: 10.66.9.95 38) [SLF4JLog:152] Negotiating protocol versionLine 4633: 2025-09-04 18:59:07-0400 DEBUG (Transport protocol: 10.66.9.95 38) [SLF4JLog:128] Local identification: SSH-2.0-http://www.sshtools.com J2SSH [CLIENT]Line 4634: 2025-09-04 18:59:07-0400 DEBUG (Transport protocol: 10.66.9.95 38) [SLF4JLog:128] EOL is guessed at CR+LFLine 4635: 2025-09-04 18:59:07-0400 DEBUG (Transport protocol: 10.66.9.9 This is very strange behaviour where you can see the Test credential works fine, but the discovery fails, stating that the credential is invalid. The error indicates that the legacy SSH client j2ssh was used to execute the SSH commands.Please follow the KB0782056 for applying the solution. 2) Shazzam (Scan) – what "good" looks like & how to fix it fast What happens Shazzam scans the IP, records open/resolved ports, and decides which port probes and classifiers to launch.It builds deviceHistoryParams (IP, DNS, ecc_queue id, etc.) so later phases can reference the same context.If no classifiable ports are found, it marks the device alive not active and can run credentialless discovery. Where to look ECC MultiProbe: UNIX – Classify (payload shows scanners for SSH, DNS, WMI/WBEM, etc.).Shazzam probe parameters (timeouts, payload size, debug) under Probe → Shazzam. Validate quickly SSH 22 shows result:"open" with a banner and portProbe:"ssh".DNS shows PTR/A in the DNS scanner block.Unopened ports list or timed_out on critical ports indicates network blocks. Common break/fix Active=false (only Alive): no classifiable open ports → open SSH/22 or adjust firewall; ensure port probe isn't suppressed by custom script.Delay WBEM confusion: if delay_wbem=true, WBEM probes run later; don't chase WBEM errors during Linux SSH triage.No MID match for port probe: verify MID selection and cluster; confirm probe Stuff exists for SSH. Do this if stuck From MID: nc -vz <ip> 22 and ssh -vvv <user>@<ip> true (no interactive prompts).Re‑run with debug on Shazzam and capture ECC payload for the case. Network Connectivity (MID → Target): Use this before and during the Scan/Shazzam phase and again for Classification (SSH). If connectivity isn't solid here, Horizontal Pattern Launcher and patterns will never fire. Step-by-step triage (run in this order) Pick the exact target & port Target IP you see in Discovery Status/ECC.SSH port (usually 22; note any custom port). Basic reachability (don't stop at ping) Linux MID: ping -c3 <ip> (ICMP may be blocked; failure ≠ proof of outage)traceroute <ip> (path, last reachable hop)ip route get <ip> (which source IP will be used; important for firewall rules/NAT) Windows MID (PowerShell): tracert <ip>Test-NetConnection <ip> -InformationLevel Detailed TCP handshake to SSH Linux MID: nc -vz <ip> 22 (or custom port) Expect: succeeded + latency.If refused: host up, service down or filtered.If timed out: network/ACL path issue. Alt: telnet <ip> 22 (if available).Windows MID: Test-NetConnection <ip> -Port 22 Banner + cipher/KEX sanity ssh -vvv -oBatchMode=yes <user>@<ip> true Expect: key exchange, auth attempt, and exit without interactive prompts.Red flags: Permission denied, no matching key exchange method, MFA/password-change prompts, or forced menu shell. DNS checks (PTR/A can affect FQDN & credential affinities) From MID: nslookup <ip> (PTR)nslookup <hostname> (A/AAAA)Mismatches/split-horizon? Capture the outputs for the case. Local firewall/proxy on the MID Linux: sudo iptables -S or sudo nft list ruleset; sudo ufw statusWindows: check Windows Defender Firewall outbound rules.Ensure the MID does not route SSH via an HTTP proxy (unset http_proxy/https_proxy for these tests). Target-side confirmation (if you have access) Check host firewall: RHEL/CentOS: sudo firewall-cmd --list-ports or iptables -L -nUbuntu/Debian: sudo ufw status Log for inbound attempts: sudo tail -f /var/log/secure (RHEL) or /var/log/auth.log (Debian/Ubuntu) while you run step 4. Packet capture (either side, if allowed): sudo tcpdump -ni any host <MID_IP> and port 22 MTU / fragmentation edge cases (WAN/VPN paths) ping -M do -s 1472 <ip> (lower size until it passes); persistent DF drops can break SSH during KEX. NAT & source IP alignment If ACLs are IP-based, make sure the firewall allows the MID's egress IP (see step 2 ip route get <ip>).Record any NAT devices on the path. ServiceNow knobs (use sparingly) If scans show frequent timed_out despite reachable hosts, temporarily raise Shazzam connect/response waits and confirm mid.discovery.max_payload_size is not constraining results. Reset after testing. Quick decision tree nc -vz = success but ssh -vvv shows prompts/mismatch → fix credentials/MFA/cipher/KEX/shell.nc -vz = refused → SSH service down or blocked on host; check target firewall/sshd.nc -vz = timed out / traceroute stalls → network/ACL/routing; involve network team with source/dest, port, and timestamps.DNS inconsistent → expect missing FQDN in results; fix PTR/A to avoid ID/FQDN surprises. Related links: 1. See the Discovery Phase Shazzam - KB0965884 to understand in detail regarding the Shazzam Phase. 2. Troubleshooting Shazzam Phase - KB0535234 3. Test MID Server connectivity 3) Classification (SSH) – prove the box is Linux & prepare pattern launch What happens SSH classifiers run small commands: uname -a, vendor checks (esx.sh, f5.sh, cloudian.sh), etc.A JavascriptProbe (Interactive Terminal) sets use_class=discovery_classy_unix, passes deviceHistoryParams, binding the credential, port, and probe ids.Successful classification produces a Linux sys_class_name and hands off to the Horizontal Pattern Launcher. Where to look ECC MultiProbe: UNIX – Classify → multiple SSHCommand entries for uname -a and vendor detectors.ECC JavascriptProbe: SSHTerminalInteractiveCommand → check use_class, port=22, credential_id, classification_probe, cidata. Validate quickly uname -a returns Linux string; no shell errors (/bin/sh or bash exists).Credential is selected (no Permission denied, no MFA/password‑change prompts).deviceHistoryParams reflects SSH open and DNS name. Common break/fix Non‑interactive prompts (MFA, expired password): exempt service account or use SSH key without passphrase.Wrong shell (csh/rbash): set login shell to /bin/bash or /bin/sh.Cipher/KEX mismatch: align server crypto with MID policy; verify OpenSSH versions.Classifier never fires: ensure Shazzam saw ssh open and triggersProbe() wasn't suppressed; verify Unix classifier is active. We get "Active, couldn't classify" when Shazzam does see a classifiable port (so the device is Active) but no classification lands—either because the credential-less rules don't match, the probe can't be launched (MID/mapping/script error), or the SSH classify probe runs but cannot authenticate/execute. Troubleshooting "Active, couldn't classify": When it occurs: Shazzam found classifiable ports (device is Active) but credential-less classification failed: After scan, Shazzam tries discovery_classy_scan. If it can't find a match, it explicitly sets the device state to Active (with "Classifying") and then to Unclassified—which surfaces in the run as "Active, couldn't classify."Shazzam intended to classify but hit setup/runtime errors: If there's no MID selection for the needed port probe, Shazzam records an error and continues; combined with (1), you end up Active→Unclassified. Likewise, any port probe script exception is caught and logged; the state still becomes Active→Unclassified.Classifier probes were launched, but OS classification via SSH didn't complete.: Shazzam does launch ClassifierProbes when the device is active; for UNIX this is an SSH classify that runs commands like uname -a. If SSH auth/exec fails (bad creds, ACL, PAM/shell restrictions), the device remains Unclassified → "Active, couldn't classify."You can see from your UNIX classify payload that SSH (22) was open with an OpenSSH banner and the UNIX classifier (use_class=discovery_classy_unix, port 22) was targeted—so a failure here would produce the symptom. Note the contrast: if no classifiable ports are found at all, Shazzam flips result.active=false, and you'd see "Alive, not active" instead—not "Active, couldn't classify." Why it occurs: No classifier match from scan evidence (banner/parameters didn't match any discovery_classy_scan rule) → Active→Unclassified.No MID server available for the relevant port probe (e.g., SSH/WBEM) → error stored; state still becomes Active→Unclassified.Port probe script error (runtime exception in a probe's script) → error logged; state Active→Unclassified.Classifier actually launched but couldn't complete (SSH): bad credentials, key mismatch, network blocks, restricted shell (can't run uname -a), or sudo elevation mis-configured—all leave the device Unclassified. Your classify probe set shows SSH 22 open with banner and the UNIX classify (SSHCommand) prepared, so failures beyond banner (auth/exec) are the likely culprits in this path. Quick triage (do these every time): Are credentials validated?Is the SSH port valid/reachable from the MID?Follow the step-by-step triage path: Confirm classifiable ports existed; otherwise you'd get "Alive, not active."Check trigger mapping: Shazzam only treats a port as "classifiable" when trigger_probe_m2m has an active mapping; verify SSH→UNIX classify mapping is present/enabled. This is what triggersProbe(...) checks.Review MID assignment/selection for that port probe; "No MID server found to classify with …" will lead to Active→Unclassified.Scan Shazzam errMsgs for port-probe script exceptions; any caught error still results in Active→Unclassified.Verify the UNIX classifier actually fired (ClassifierProbes.launch() runs for Active devices discovering CIs). If it fired, inspect the SSH probe results/ECQ payload for auth/command errors. Related ERRORS: Connection issue with discovering a particular Linux server, which is failing in the Classification phase with "Active, couldn't classify". The warning states:Cannot connect, status is SSH_CONNECTION_FAILURE. Could not agree on signature algorithm Client: [ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss] Server: [] This will be caused by the Permission issue on the host key file, ssh_host_ecdsa_key. Please follow the KB1425502 for the resolution. Discovery doesn't classify Linux servers, throws Active, couldn't classify: KB0690040 Linux Discovery failing with signature algorithm error [ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521, rsa-sha2-256, rsa-sha2-512, ssh-rsa, ssh-dss] Server: []: KB1277547 Troubleshooting the Classification Phase in Discovery: KB0535236 Discovery Troubleshooting Guide - Protocol Level: KB0868224 4) Horizontal Pattern Launcher – Linux Server (+ extra patterns) What happens A HorizontalDiscoveryProbe called "Pattern Launcher: Linux Server" fires with: patternId for Linux Server, sys_class_name=cmdb_ci_linux_server, protocol=SSH, port=22.credAffinity → which SSH credential id is bound.triggered_probes → list of pattern probes to run (includes Linux Server and, when detected, additional patterns such as UNIX Cluster – VERITAS).Optional excludeSshInteractivePatterns to skip interactive DB/flow patterns. Where to look ECC HorizontalDiscoveryProbe – Pattern Launcher: Linux Server. Validate quickly pattern is Linux Server and the patternId matches content version.triggered_probes includes the expected Linux steps; additional patterns may appear (we don't explain them here—just confirm the process picked them up).credAffinity points to a valid SSH credential. Common break/fix Launcher appears but nothing runs: check MID outbound → ECC, and watch for errors in HorizontalDiscoverySensor (next phase).Unexpected additional pattern: that's OK; launcher can chain others (e.g., VERITAS). Only investigate if they fail noisily. 5) Pattern execution (Linux) – what the probes do Typical Linux pattern probes you'll see: Processes snapshot: ps awwxo pid,ppid,command | sed -n '/<defunct>/!p' (with filters for sd‑pam, NetworkManager, automount, etc.).Open sockets: sudo lsof -iTCP -n -P -F pcnfT (requires sudo NOPASSWD).Installed software: an RPM/DPKG "pkg_munger" bash that prints Package/Status/Maintainer/Version/Release (fallback to Debian's /var/lib/dpkg/status). Validate quickly sudo is non‑interactive; commands return output (no "sudo: a password is required").For software: RPM present or DPKG fallback used. Common break/fix sudo blocked: add NOPASSWD for dmidecode,lshw,fdisk,dmsetup,multipath,lsof (+ any site‑specific).Minimal hosts missing tools: install iproute2, lsof, dmidecode, etc., or adjust pattern expectations.Large payloads trimmed: increase MID/Probe payload size or reduce scope. Tip: when Horizontal Logs show a failure, match the Pattern → Step # / Step name below, then run the "Quick check" on the target. Linux – Find FQDN (pattern id 5e1243e39f2032001d753758442e7041) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Run FQDN script fileRuns linux_fqdn.sh; expects a single line FQDN (excludes lines with "Unable").Script present but returns blank or non-FQDN text.Run the same script as MID: bash linux_fqdn.sh; ensure DNS returns an FQDN; fix /etc/hosts/DNS.2Validate FQDN formatRegex-tests FQDN; rejects invalid format.Host only (no dots) or trailing dot.hostname -f; ensure proper domain search; repair DNS or /etc/hostname.3Update the Linux CI FQDNWrites cmdb_ci_linux_server.fqdn if FQDN non-empty.FQDN empty from prior steps.Fix Steps 1–2; re-run Horizontal Pattern. Linux – Distribution (pattern id fe32c7e39f2032001d753758442e7065) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Get distribution`cat /etc/*releasegrep -v ID_LIKEgrep -v LOGO`; expects release content.2Extract distribution versionJS parses known vendor/version strings.Output locale/format differs from regex.Print file and compare against regex variants; widen regex or upgrade content pack.3Extract OS distributionMaps vendor to normalized OS name.Vendor strings unexpected.Confirm strings; patch mapping; ensure /etc/*release includes vendor.4–5Update OS / Update OS version on Linux CIWrites to CI fields if values present.Upstream attrs empty.Fix Steps 1–3; re-run. Linux – Hardware Information (pattern id c161cfa39f2032001d753758442e70f9) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Set the privileged command to usePrepares sudo wrapper.N/A (sets context).Ensure sudoers NOPASSWD on required cmds.2Get DMI type 1`sudo dmidecode -t 1cat`; expects Manufacturer/Serial/UUID/Product.DMI locked/garbled ("To Be Filled By O.E.M.") or no sudo.3Failover Get DMI type 1Same command, simpler parsing.Output present but parser fields missing.Verify raw text includes keys; consider lshw fallback in env.4Convert output to JSONParses text into JSON when table empty.Upstream output_dmi empty.Fix Steps 2–3; then proceed.…Ref/Rel Linux SerialCreates serial↔CI relation.Missing serial.Ensure DMI provides serial; else vendor tooling. Linux – Network (pattern id 0ae107e39f2032001d753758442e70ec) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Check kernel version`uname -rcut -d'.' -f1`; used to branch.N/A2Get Interfaces (if use_net_tools ≠ false)`ifconfig -aawk …intointerfaces` table.net-tools absent; awk col shifts.3Construct network Adapter Exclusion regExpBuilds regex from property.Empty property leads to no-op (not a failure).Populate property if needed.4Construct NIC and IP Map for Exclusion List of AdaptersJS parses interfaces text; extracts IPs with regex.Text format variations (BusyBox, bonding).Validate regex on host; adjust exclusions.5Failover: Get InterfacesCustom ip addr parser to interfaces_ss.Old ip output/locale; grep filtered too much.ip addr raw; remove greps; set LC_ALL=C.6Union interfaces and interfaces_ssMerges both sources.Both empty ⇒ downstream empty.Ensure either step 2 or 5 returns data.7Get interface names & Mac - for kernel 3 and above`ip -o link showawk …` → (name, mac).AWK fields differ across kernels.8Populate interface names without @ - for kernel 3 and aboveCleanup names (remove @).N/AN/A. Linux – Network ARP Tables (pattern id 84538fe39f2032001d753758442e7086) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Get ARP details using iproute2`ip nawk '{print $1,$3,$5}'` → (ip, ifName, mac).Empty neighbor cache; fields differ by kernel.2Filter ARP table with empty valuesDrops rows missing IP/ifName/MAC.Step 1 produced partials.Fix Step 1; warm cache.3Get ARP (fallback)arp -n parse (ip, mac, ifName).net-tools missing; "incomplete" lines filtered out → empty.arp -n; install net-tools or rely on Step 1.4Update discovery_net_arp_tableMaps columns to discovery table.Upstream empty.Fix Steps 1–3.5Remove duplicate Network ARP TablesJS de-dup on (ip, mac).N/AN/A6Ref/Rel between discovery_net_arp_table and Linux CICreates relations; requires rows.No rows from prior steps.Ensure Steps 1–4 return rows. Linux – CPU (pattern id e69343279f2032001d753758442e7067) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Get CPU infocat /proc/cpuinfo (table lines).Container/minimal images with trimmed proc entries.cat /proc/cpuinfo on target.2Get flags datagrep hypervisor /proc/cpuinfo.No "hypervisor" flag even if VM (older kernels).Tolerate empty; do not fail pipeline.3Check if is virtualSets boolean from Step 2.Assumes presence of flag.Treat as heuristic only.4Filter CPU dataKeeps only relevant lines.Localized headers (non-English).LC_ALL=C cat /proc/cpuinfo.5Get thread corelscpu and parse "Thread" line.lscpu missing/old; localized output.Install util-linux; set LC_ALL=C. Linux – Memory (pattern id 4103cbe39f2032001d753758442e709a) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Get memorygrep MemTotal /proc/meminfo → value in KB at column 2.Non-standard /proc/meminfo or grep regex mismatch.grep MemTotal /proc/meminfo.2Update RAM on Linux CIConverts to MB and writes.Upstream empty.Fix Step 1. Linux – Memory Modules (pattern id dc334fe39f2032001d753758442e70b7) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Get memorygrep MemTotal /proc/meminfo → value in KB at column 2.Non-standard /proc/meminfo or grep regex mismatch.grep MemTotal /proc/meminfo.2Update RAM on Linux CIConverts to MB and writes.Upstream empty.Fix Step 1. Linux – Storage (pattern id 303407279f2032001d753758442e708c) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Run storage script fileExecutes storage.bash; expects marked sections.Script present but returns nothing.Run script; check environment (PATH/sudo).2–3Get dmsetup ls / Parse dmsetup lssudo dmsetup ls; parse alias→(major:minor).No device-mapper present or permission denied.sudo dmsetup ls; grant NOPASSWD.4–5Get dmsetup table / Parse dmsetup tableMaps extents & multipath from table.Output empty on hosts without DM; treat non-fatal.sudo dmsetup table.6Get FC dataParses "fc" section from script output.Section absent; should be non-fatal.Check HBA presence.7–8Remove new lines / Map output to collectionNormalizes and parses multiple sections.Upstream stOut empty.Fix Step 1.9Get vxvmcommand -v vxprint.VxVM not installed (expected blank).N/A (non-fatal).10+proc_ide/sys_block tablesBuild device tables from parsed sections.If sections missing, tables empty.Ensure script sections exist. Linux – Cloud (pattern id 4660f86edb057200c12ef9361d96190c) This pattern includes AWS and Azure sub-patterns. Linux – AWS (pattern id 5540b86edb057200c12ef9361d9619f6) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Check if it is compiled by amazonIMDSv2 token PUT (HTTP 200) ⇒ amazon.IMDS blocked or no route.curl -sS -o /dev/null -w '%{http_code}\n' -X PUT 'http://169.254.169.254/latest/api/token' -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' (expect 200).2–5Failover: Check if it is compiled by amazon/sys/hypervisor/..., dmidecode -s bios-vendor (with/without sudo).No sudo; DMI not exposed in cloud image.sudo dmidecode -s bios-vendor; grant NOPASSWD.6If compiled by amazon, mark it as EC2 serverSets EC2 attributes.Prior steps didn't confirm.Open IMDS; or relax detection to additional hints. Linux – Azure (pattern id 3f7e3e18db5932003398f1351d9619e5) Step #Step name (exact)What it runs / expectsTypical "status 0 fail" causeQuick check / Fix1Set the privileged command to usePrepare sudo.N/AEnsure sudoers.2Create isAzure variableInitializes state.N/AN/A.3Search for unknown-245 optionGreps DHCP leases for option-245.Different lease paths or interface names.Adjust grep paths; check /var/lib/dhclient/*.leases.4Set isAzure variable if option-245 is enabledMarks Azure if found.N/AN/A.5Check waagent version - no pathwaagent -version w/o path.waagent missing or not in PATH.Try /usr/sbin/waagent -version; install WALinuxAgent. Linux – Memory / Uptime quick refs UNIX – OS Uptime (pattern id 80d2cbe39f2032001d753758442e7092): Step 1/2/3 parse variants of uptime output; failures are typically regex vs locale differences — set LC_ALL=C and re-test.Linux – Memory already above (Steps 1–2). 6) Horizontal Discovery Sensor – turning pattern output into CI data What happens Sensor reads pattern JSON; if empty or malformed, it stops and logs a clear error.It builds a HorizontalDiscoveryResultHandler, runs Pre‑Process hooks (including Pre Payload Processing Scripts), and only then calls IRE.It may drop IPs for hardware CIs to prevent IP flipping, and can cache cloud resource counts.It then handles references/relations unless using "internal_id based payload" (then IRE owns it). Validate quickly No "payload output is empty/unexpected format" errors.If graceful terminate appears, the pattern ended by design—use Pattern Log to see why.Look for "Identification engine errors" → jump straight to Identification Logs. Common break/fix Empty/malformed payload: run Pattern Debug; check failed step; Pre‑processing script failed: disable custom pre‑scripts or fix errors; rerun with debug.Check the Identification and Reconciliation rules are OOTB, and if not always a good step to revert to OOTB and re-run the discovery See the KB0535238 Troubleshooting the Identification Phase in Discovery 7) Identification & Exploration (IRE) – creating/updating the CI What happens Discovery uses CMDB Identification Engine (modern path) to decide insert vs update.Prevents host‑name/IP flipping before commit; logs ID attempts.On re‑classification blocks, a Reclassification task is raised.Handles duplicate IPs and locking; then reconciles related lists and triggers supplementary classifiers. Validate quickly Check Identification Logs (from Pattern/Sensor log link).Device history shows Identified CI (or "Identified, not updating CI" / "Identified, ignored extra IP"). Common break/fix LOCK FAILED / duplicate IP: wait for lock or stop concurrent runs on the same host/IP set.Reclassification not allowed: use the generated task/workflow to approve class change.No matchable info: ensure pattern collected minimal identifiers (name, serial, MAC/IP). Common Errors: Found multiple non-dependency relations [parent:'PARENT_CI_SYSID'/child:'CHILD_CI_SYSID'/type:'RELATIONSHIP_TYPE_SYSID'] between payload items. In case the identification found multiple relationships between the same application and the child CI. There should only be one. See the KB0749044: "MULTI_MATCH" and message "Found multiple relations" PART III — Decision trees A) "Could not classify" (SSH) From MID: nc -vz <ip> 22 → open? If no, fix firewall/route.ssh -vvv <svc>@<ip> true → any MFA/expiry/menu shell? Fix policy or switch to key.In ECC UNIX – Classify, confirm uname -a ran and deviceHistoryParams exists.In ECC SSHTerminalInteractiveCommand, verify use_class=discovery_classy_unix, port=22, credential_id.Re‑run; check Horizontal Pattern Launcher appears. If not, review classifier record (active? conditions?). B) "Pattern didn't start" after classification ECC HorizontalDiscoveryProbe → Pattern Launcher: Linux Server present? If no, classifier/policy issue.If present, confirm credAffinity and triggered_probes list.Check ECC MultiProbe/SSHCommand pattern steps appear after launcher.If missing, check ECC queue errors and MID → instance connectivity. C) "Pattern ran but CI not updated" ECC HorizontalDiscoverySensor shows payload empty/unexpected? Fix failing step via Pattern Debug.If Identification engine errors, open Identification Logs (link in sensor log) and resolve identifiers.Watch for IP flipping prevention—make sure proper primary IP is in payload. D) "sudo keeps prompting / storage steps fail" Grant NOPASSWD for required tools; confirm which dmidecode dmsetup multipath fdisk lsof returns binaries.Validate PATH in non‑interactive SSH (/usr/sbin included). PART IV — Using the OOTB Pattern Debug (Horizontal logs) Run → Single IP with Pattern Debug enabled. Open Discovery Status → Pattern Log to see each node, command, and parsed output.Cross‑reference the same run's ECC records via the agent correlator/ECC breadcrumbs to match HorizontalDiscoveryProbe → MultiProbe/SSHCommand → HorizontalDiscoverySensor.For ID issues, follow the link to Identification Logs from the sensor warnings.If a step fails silently, temporarily raise MID probe log level and rerun on a single IP. PART V — What "good" data looks like cmdb_ci_linux_server exists with name/host_name, os_version and kernel_release.RAM/CPU approximate host reality.NICs, IPs, and FQDN present when DNS PTR exists.Software list populated (RPM/DPKG).Relations updated (e.g., IP ↔ NIC, mounted filesystems).If clusters/DB detected, additional pattern CIs/relations exist. PART VI — Intake Template (for escalations) IP(s)/Range, Run # / Status sys_id, CorrelatorMID(s) (name, version, cluster, health)Credential(s) (type, test result, sudo policy)Target OS (distro/version, shell)Network/DNS (ACLs, proxies, split‑horizon)ECC samples (Shazzam Classify, SSHTerminalInteractiveCommand, Pattern Launcher, MultiProbe step, HorizontalDiscoverySensor)Pattern Debug & ID Logs excerptsPRB/Store state Appendix A – Target‑side quick commands (run via SSH if needed) # Identity & basicsuname -a; id; echo $SHELL; getent passwd $(whoami) | cut -d: -f7# Networkip -o addr; ip route; getent hosts $(hostname -f)# DNShost $(hostname -f) || nslookup $(hostname -f)# Sudo sanity (no prompt should appear)sudo -n true; echo $?# Tools needed by patternswhich ip lsof dmidecode dmsetup multipath fdisk || echo missing# Software inventoryrpm -q -a | head || cat /var/lib/dpkg/status | egrep '^Package:|^Status:|^Version:' | head Appendix B – Suggested sudoers (tune paths per distro) <svc_account> ALL=(root) NOPASSWD: /sbin/dmidecode, /sbin/lshw, /sbin/fdisk, /sbin/dmsetup, /sbin/multipath, /usr/sbin/lsof Appendix C – Reading ECC like a pro Topic tells you the actor: Shazzam/MultiProbe/SSHCommand/JavascriptProbe/HorizontalDiscoveryProbe/HorizontalDiscoverySensor.Response to links the previous step; follow it forward to see the full chain.Payload params to note: use_class, deviceHistoryParams, cidata, credAffinity, patternId, triggered_probes, port/protocol, glide.xmlhelper.trim.enable, ecc_breadcrumbs. Appendix D – PRB/Store checklist Linux pattern version vs instance package levelChanges in iproute2/net-tools availabilityCloud metadata/IMDS shiftsIdentifier changes affecting IRE (reclassification rules) Always gate yourself: (1) Credential validated? (2) SSH port valid? (3) Followed triage flow end‑to‑end? (4) PRB/Store checked? Related Links<!-- /*NS Branding Styles*/ --> .ns-kb-css-body-editor-container { p { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } span { font-size: 12pt; font-family: Lato; color: var(--now-color--text-primary, #000000); } h2 { font-size: 24pt; font-family: Lato; color: var(--now-color--text-primary, black); } h3 { font-size: 18pt; font-family: Lato; color: var(--now-color--text-primary, black); } h4 { font-size: 14pt; font-family: Lato; color: var(--now-color--text-primary, black); } a { font-size: 12pt; font-family: Lato; color: var(--now-color--link-primary, #00718F); } a:hover { font-size: 12pt; color: var(--now-color--link-primary, #024F69); } a:target { font-size: 12pt; color: var(--now-color--link-primary, #032D42); } a:visited { font-size: 12pt; color: var(--now-color--link-primary, #00718f); } ul { font-size: 12pt; font-family: Lato; } li { font-size: 12pt; font-family: Lato; } img { display: ; max-width: ; width: ; height: ; } } Troubleshooting Shazzam Phase: KB0535234Troubleshooting the Classification Phase in Discovery: KB0535236Troubleshooting the Identification Phase in Discovery: KB0535238Troubleshooting the Exploration Phase in Discovery: KB0535240Discovery troubleshooting | Error messages: KB0539839