Discovery Troubleshooting Error Codes



Discovery Troubleshooting

This page captures the remediation steps we use for troubleshooting the Discovery Logs, generated by our Discovery Troubleshooting Application. It is the precursor to understanding the Root Cause and Remediations on this page and is created based on the analysis of the Discovery Logs.
Bookmark this page as we constantly update this knowledge base to iteratively improve how we do Discovery Troubleshooting.

Each row in the Report corresponds to an IP Address associated with a Device.
The Report will have one or more of the following columns:
1. Discovery Status: Instance of the Discovery Schedule in ServiceNow corresponding to the analysis.
2. IP Address: IP Address of the Device.
3. DNS Name: DNS Name returned by Discovery.
4. Protocol: WMI, SSH or SNMP.
5. Error Code: Mapping to the Error Codes on this page, to capture Root Cause and Remediation.
6. Location: The location of the Discovered CI, driven by the Discovery Schedule.

The sections below detail the Team Responsible for the Troubleshooting, the Root Cause and the Remediation for the Error Code(s) captured in the Report.


Error Codes


P0. Phase 0: No Ping
      [ P0.ALL.01 | P0.ALL.02 | P0.ALL.03 ]
P1. Phase 1: Port Scan
      [ P1.ALL.01 ]
P2. Phase 2: Classification
      [ P2.WMI.01 | P2.WMI.02 | P2.WMI.03 | P2.WMI.04 ]
      [ P2.SSH.01 | P2.SSH.02 | P2.SSH.03 | P2.SSH.04 ]
      [ P2.SNMP.01 | P2.SNMP.02 | P2.SNMP.03 | P2.SNMP.04 ]
      [ P2.WBEM.01 ]
P3. Phase 3: Identification
      [ P3.ALL.01 | P3.ALL.02 | P3.ALL.03]
      [ P3.SSH.01 | P3.SSH.02]
      [ P3.SNMP.01]
P4. Phase 4: Exploration
      [ P4.WMI.01 ]
      [ P4.SSH.01 | P4.SSH.02 | P4.SSH.03 ]
      [ P4.SNMP.01 | P4.SNMP.02 ]


P0. Phase 0: No PingTop


P0.ALL.01:
Team Responsible: ServiceNow Support Team.
Root Cause: IP Address associated with the Device is not a part of the IP Range(s) currently configured in the Discovery Schedule(s).
Remediation: Update the IP Range corresponding to this IP Address in the appropriate Discovery Schedule.

P0.ALL.02:
Team Responsible: Device Support Team.
Root Cause: Device does not exist on the Network, or Device is switched off, or Device is not responding to Ping due to malfunction.
Remediation: Ensure device is on the Network, is switched on and functioning correctly.

P0.ALL.03:
Team Responsible: Network Support Team.
Root Cause: Device exists on the Network, is switched on and functioning correctly, but there is a firewall between the Device and the ServiceNow MID Server blocking the ICMP Protocol.
Remediation: Troubleshoot and remediate firewall configuration (Routes and Rules) for ICMP Protocol between Device and MID Server.


P1. Phase 1: Port ScanTop


P1.ALL.01:
Team Responsible: Device Support Team.
Root Cause: Device is not accepting connections on the port being probed by the Shazzam Port Probe.
Remediation: Troubleshooting steps should be followed in the order given below.
1. Enable the Device to accept connections on the port being probed by the Shazzam Port Probe.
2. Work with the Network Team to update Network Firewall rules / policies to allow WMI / SSH / SNMP traffic between the MID Server and the Device.
3. Device has non-standard port configurations. It is not being detected by the out-of-the-box ServiceNow Shazzam Probe. Work with the ServiceNow Team to configure a custom Port Probe with the applicable IP Services, if Device is in scope for Discovery.
4. If this is an SNMP Device then: Enable SNMP on the Device AND Configure ACL on SNMP Device to allow connections from IP Addresses of all the configured ServiceNow MID Server Hosts AND Validate the Device Credentials stored in ServiceNow.


P2. Phase 2: ClassificationTop


P2.WMI.01:
Team Responsible: Device Support Team.
Root Cause: Windows Credentials are not present or incorrect.
Remediation: Troubleshooting steps should be followed in the order given below.
1. Verify if the Windows Domain Credentials (Domain\Username) are stored in ServiceNow.
2. Verify if the same Credentials are configured correctly on the Device.
3. Verify that the credentials have access rights to the Servers. The credentials should be a Domain User with Local Admin Privileges.

P2.WMI.02:
Team Responsible: Device Support Team.
Root Cause: Windows credentials populated in the Service Now Credential Table, but are not being used and instead MID Server service account is used for authentication on the device.
Remediation: As a best practice MID Server service account should be avoided while discovering Windows Devices. This can be achieved by disabling the 'mid.powershell.local_mid_service_credential_fallback' configuration in MID Server properties. However, this requires PowerShell to be installed on MID Server Host and applicable Windows Credentials configured in the ServiceNow Credentials Module. This error can be ignored if it appears in conjunction with P2.WMI.01 or P2.WMI.04 on the same IP Address / Windows Device.

P2.WMI.03:
Team Responsible: Device Support Team.
Root Cause: WMI Access is blocked, preventing remote WMI queries from being executed on the Device.
Remediation: Troubleshooting steps should be followed in the order given below.
1. Confirm that the Windows Domain User (configured in ServiceNow) has Local Admin privileges on the Device.
2. If Windows Firewall is enabled, confirm that the Windows Firewall Rules allow the Windows Domain User to run remote WMI queries on the Device. If permitted, simply disable the Windows Firewall.
3. If another Software is installed (example: Symantec, McAfee, Check Point, etc), confirm that its settings allow the Windows Domain User to run remote WMI queries on the Device.

P2.WMI.04:
Team Responsible: Device Support Team.
Root Cause: DCOM Ports are blocked on Device.
Remediation: Open DCOM Ports on Device.
About DCOM: DCOM (Distributed Component Object Model) is a framework used by Windows to allow COM components to work over the network. Unlike your traditional TCP/IP and UDP/IP services where a single protocol has a fixed port DCOM dynamically assigns ports for the COM objects it remotes.
WMI and Powershell access remote Windows machines on port 135 (RPC). For remote Windows machines where Windows Firewall is enabled, it is not enough just opening port 135 to have Discovery successfully discover the machine.
When the MID server knocks at the door of RPC on the remote server via 135, the remote machine announces a (dynamic) port that the MID server has to use in order to access DCOM back on the remote server. The port announced by the RPC Server on the remote machine is one of a range of ports (unless RPC is configured to run with a static port).
The default ranges of DCOM ports are:
from 1025 to 5000: Windows 2000, Windows XP and Windows Server 2003
from 49152 to 65535: Windows Server 2008 and later versions, and in Windows Vista and later versions



P2.SSH.01:
Team Responsible: Device Support Team.
Root Cause: SSH Credentials are not present or incorrect.
Exception for SNMP Devices responding to SSH Protocol: Refer to Error Code P1.ALL.01, Bullet Number 4.
Exception for ESXi Hosts responding to SSH Protocol: Ignore this error as these devices are classified and populated via vCenter Discovery for ESXi Devices.
Remediation: Troubleshooting steps should be followed in the order given below.
1. Verify if the Device Credentials stored in ServiceNow.
2. Verify if the same Credentials are configured correctly on the Device.
3. Verify that the credentials have access rights to the Servers. The credentials should have Root privileges or preferably have the right to execute certain commands with Root Privileges using sudo.
Ignore error if this is a Network Device or ESXi devices, as these devices are classified and populated by SNMP Discovery for Network Devices and vCenter Discovery for ESXi Devices.

P2.SSH.02:
Team Responsible: Device Support Team.
Root Cause: Connection closed by server.
Remediation: Allow the authorized access/connection from the MID Server to Device. Ignore error if this is a Network Device as these devices are classified and populated by SNMP Discovery.

P2.SSH.03:
Team Responsible: Device Support Team.
Root Cause: Connection temporary closeout due to multiple probing.
Remediation: Typically, these devices are discovered on different IP Address or in the next discovery run. This error can be ignore, unless it is consistently recurring.

P2.SSH.04:
Team Responsible: ServiceNow Support Team.
Root Cause: SSH Devices responding to SNMP.
Remediation: To avoid discovery of SSH device via SNMP classification, add correct ssh credentials for that device in ServiceNow and disable respective SNMP Classifications for such devices. If these are SNMP Devices responding to SSH only, but are not responding to SNMP, then refer to P1.ALL.01 for remediation.

P2.SSH.05:
Team Responsible: Device Support Team.
Root Cause: Connection closeout due to wrong credentials, or access is blocked.
Remediation: Verify and update the credentials in ServiceNow or troubleshoot the reason for why the probes are being blocked.



P2.SNMP.01:
Team Responsible: ServiceNow Support Team.
Root Cause: SNMP OID not present in ServiceNow.
Remediation: Add the SNMP OID of the Device to the SNMP Classifier associated with it, in ServiceNow (if Device is in scope for Discovery).

P2.SNMP.02:
Team Responsible: ServiceNow Support Team.
Root Cause: SNMP Classifier not configured in ServiceNow.
Remediation: Configure the SNMP Classifier in ServiceNow and associate the SNMP OID of the Device to it, (if Device is in scope for Discovery).

P2.SNMP.03:
Team Responsible: ServiceNow Support Team.
Root Cause: SNMP Classifier not configured or configured incorrectly in ServiceNow.
Remediation: Configure a new SNMP Classifier or fix the existing SNMP Classifier, (if Device is in scope for Discovery).

P2.SNMP.04:
Team Responsible: Device Support Team.
Root Cause: Credential Error preventing Device to return any meaningful data to ServiceNow Probe.
Remediation: This could a be false-positive error, if the Device is busy while being discovered. Validate the Credentials on the SNMP Device by doing a targeted Discovery via "Discover Now" Related Link. If Discovery consistently fails, validate credential on SNMP Device and ServiceNow.

P2.SNMP.05:
Team Responsible: Device Support Team.
Root Cause: ACLs preventing Device to return any meaningful data to ServiceNow Probe.
Remediation: Validate the ACLs on the SNMP Device. Minimum data needed to be returned (via ECC Queue) to classify SNMP Devices is sysName along with sysObjectID and/or sysDescr.



P2.WBEM.01:
Team Responsible: Device Support Team.
Root Cause: CIM Credentials not present or incorrect (CIM, CIM-XML, CIM Query Language).
Remediation: Ignore error if it is ESX and ESX Appliances. These are discovered via the vCenter, by leveraging credentials VMware Credentials.


P3. Phase 3: IdentificationTop


P3.ALL.01:
Team Responsible: Device Support Team.
Root Cause: Same device is found via another IP Address.
Remediation: Ignore error.

P3.ALL.02:
Team Responsible: ServiceNow Support Team.
Root Cause: Duplicate records exist in the CMDB.
Remediation: Remove duplicate data from the CMDB (based on the Identifier) and / or fix the Identifier in ServiceNow for that Device Type.

P3.ALL.03:
Team Responsible: ServiceNow Support Team.
Root Cause: Duplicate records exist in the Serial Number Table.
Remediation: Remove duplicate data from the Serial Number Table and / or fix the Identifier in ServiceNow for that Device Type.



P3.SSH.01:
Team Responsible: Device Support Team.
Root Cause: Permission not granted to run 'dmidecode' command on Linux Server.
Remediation: Provide sudo permission to the MID Server User to successfully run the 'dmidecode' command.

P3.SSH.02:
Team Responsible: Device Support Team.
Root Cause: Permission not granted to run 'sneep' command on Solaris Server.
Remediation: Provide permission to the MID Server User to successfully run 'sneep' command to discover chassis serial number of Solaris Server.



P3.SNMP.01:
Team Responsible: Device Support Team.
Root Cause: ACL on SNMP Device is preventing to extract identifiable information during Identification phase. Since the device is classified in previous phase, it should generally pass Identification also. This happens mostly if device is busy on active probing on alternate IP address.
Remediation: Ignore this error as the device will get discovered on another IP address linked to it or in the next discovery run.


P4. Phase 4: ExplorationTop


P4.WMI.01:
Team Responsible: ServiceNow Support Team.
Root Cause: ServiceNow Service ID does not have permissions to execute specific WMI commands configured in the Probe.
Remediation: Troubleshooting steps should be followed in the order given below.
1. Further troubleshoot error to determine which commands do not have permissions.
2. Work with Application Support Team to grant access to ServiceNow Service ID to execute those commands on the Device, if specific attributes returned by those commands are in scope of the CMDB Design.



P4.SSH.01:
Team Responsible: ServiceNow Support Team.
Root Cause: ServiceNow Service ID does not have permissions to execute specific SSH commands configured in the Probe.
Remediation: Troubleshooting steps should be followed in the order given below.
1. Further troubleshoot error to determine which commands do not have permissions.
2. Work with Server Support Team / Application Support Team to grant access to ServiceNow Service ID to execute those commands on the Device, if specific attributes returned by those commands are in scope of the CMDB Design.

P4.SSH.02:
Team Responsible: Device Support Team.
Root Cause: ServiceNow Service ID does not have permissions to execute SSH command 'dmidecode' configured in the Probe, thus preventing attributes like Serial Number, Model and Manufacturer from being discovered.
Remediation: Device Support Team to verify and grant access to ServiceNow Service ID to execute the 'dmidecode' command on the Device.

P4.SSH.03:
Team Responsible: Device Support Team.
Root Cause: ServiceNow Service ID does not have permissions to execute SSH command 'abd' configured in the Probe, thus preventing attributes like CPU Speed and Memory from being discovered.
Remediation: Device Support Team to verify and grant access to ServiceNow Service ID to execute the 'abd' command on the Device.



P4.SNMP.01:
Team Responsible: Device Support Team.
Root Cause: ACL on SNMP Device is preventing ServiceNow MID Server to retrieve additional attributes from Device.
Remediation: Configure ACL on SNMP Device to allow retrieval of attributes in scope of the CMDB Design.

P4.SNMP.02:
Team Responsible: ServiceNow Support Team.
Root Cause: MIB for the SNMP Device is not configured in ServiceNow.
Remediation: Configure MIB for the SNMP Device in ServiceNow and configure SNMP Walk in ServiceNow to retrieve attributes in scope of the CMDB Design.




For more information, contact us at info@quicknexus.com