Host Not Reporting Data

Follow The below steps to troubleshoot any HRND alerts.

Server not reachable

If the server is not reachable, or no login prompt on ssh, or connection timed out on ping etc, Reach out to the On-call for support.

For Azure Servers, Try to connect through the serial console in Azure Portal.
For VMWare, Try to conect to the console through the VCenter. [If still facing issues, connect with VMWare team for help.]

If none of the troubleshooting seem to work, connect with the SM Team and request a bridge call, Bring in the SME (Subject Matter Expert) and POC (Point Of Contact) and check what needs to be done to resolve the issue.

Server is reachable

1. Check Splunk status.

/opt/splunkforwarder/bin/splunk status

2. Check Telnet for the below IP:PORT, Port should be opened and connected.

Deployment Server
```
telnet 10.113.2.22 8089
```

Indexer Servers

telnet 10.113.2.15 9997

telnet 10.113.2.16 9997

telnet 10.113.2.17 9997

3. Check Server.conf, Verify the host name is correct inside. Else correct it

cat /opt/splunkforwarder/etc/system/local/server.conf

server.conf
[general]
serverName = <host-name>
...
...

4. Check Inputs.conf, Verify the host name is correct inside. Else correct it.

cat /opt/splunkforwarder/etc/system/local/inputs.conf

If the file is not present, create the inputs.conf file and add the stanza as shown below.

vi into the inputs.conf file

vi /opt/splunkforwarder/etc/system/local/inputs.conf

Copy paste the below and change the hostname accordingly.
inputs.conf
1 2
[default] host = <host name>
Save and Exit using ESC and :x
Change directory to local directory of splunk.
```
cd /opt/splunkforwarder/etc/system/local/
```
Run the below command, to change owner of splunk.
```
chown -R splunk:splunk inputs.conf
```
If splunk user is not present, then
```
chown -R root:root inputs.conf
```

Then, Restart Splunk.

/opt/splunkforwarder/bin/splunk restart

5. Check Deployment.conf, Verify the deployment server (10.113.2.22) is set correctly or not.

cat /opt/splunkforwarder/etc/system/local/deploymentclient.conf

If the deployment server is set incorrectly or not set, Use below

/opt/splunkforwarder/bin/splunk set deploy-poll 10.113.2.22:8089

6. Check Apps, Verify the apps are present or not (you should see folders for collecting metrics data such as infra metric, process, services etc.).

ls /opt/splunkforwarder/etc/apps/

If not present, contact cluster team.

7. Check Splunk error logs

tail -500 /opt/splunkforwarder/var/log/splunk/splunkd.log | grep -i error

8. Check Time and NTP configuration, Verify whether the problematic server is in sync with NTP server. Verify the time in both NTP server and the problematic server and it should be same

timedatectl status

Important

If all the above checks are passed and nothing remains to check, raise a request to network team and capture the traffic between the problematic server to the INDEXERS. Probably firewall or any rules could be blocking the communication.