- Newest
- Most votes
- Most comments
Hi,
To diagnose your issue start by checking whether everything server side is configured correctly, you can use the netstat command to verify that your application (spluk) is listening on the right port, see sample:
$ netstat -tulpn
Output
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
The above outputs should list your application and the port it is listening to for connections, for my example we can see that I have port 22 listening tcp6. Thereafter to confirm if the issue is with my security group I will use telnet or netcat to test connectivity, in my case I prefer netcat, see my samples below:
- When port 22 is open on security groups
$ nc -v 3.95.254.52 22
Output
Connection to 3.95.254.52 22 port [tcp/ssh] succeeded!
SSH-2.0-OpenSSH_7.4
- When port 22 is closed on security group
$ nc -v 3.95.254.52 22
Output
nc: connect to 3.95.254.52 port 22 (tcp) failed: Connection refused
You might also see a connection timed out on ports closed on security groups, I should further mention that your instance within the same VPC communicate over private IPs it might be beneficial to test and ensure that internal communication works.
I am aware that splunk utilizes two network interfaces one for management and the other communication with your environment, however I am not well versed with the application. I have sourced this article which is a good reference for deploying a Splunk environment on AWS, might be beneficial to review.
Else I would say let's start by isolating our issue to OS or Network config and work from there, feel free to keep me posted on your progress on this forum should you still require assistance.
It would be good to isolate whether the problem is with the AWS network or the EC2 Linux configuration.
You can verify the reachability of your AWS network configurations, such as RouteTable, SecurityGroup, NACL, and other else, with the VPC Reachability Analyzer.
Troubleshoot VPC connectivity issues with Reachability Analyzer
Relevant content
- asked 3 years ago
AWS OFFICIALUpdated 4 years ago- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 10 months ago
- AWS OFFICIALUpdated 2 years ago
Thank you so much for the response to my question I ran netcat got the following back:
splunk@Indexer2:/opt$ nc -v 172.31.1.6 9997 nc: connect to 172.31.1.6 port 9997 (tcp) failed: Connection timed out splunk@Indexer2:/opt$
Afterward I double-checked the VPC to make sure all of my instances were configured on the same VPC which they were. Should I create a new VPC and if so is there a way to edit my current instances to be applied to that new VPC if need be?
Thanks for the feedback,
Based on the outputs it does appear that your instance is not getting a response from the server, this is due to connection timed out. The error indicates that the server didn't respond to the client and the client program gave up (timed out). This could be due to:
-- The security group or network ACL doesn't allow access. -- There's a firewall on the instance's operating system. -- There's a firewall between the client and the server. -- The host doesn't exist.
There is two things you need to confirm here:
To further answer your additional questions, unfortunately you won't be able to move the instances between VPCs, what you can do is the following,
You can also check this knowledge article so basically this article covers the "connection timed out" from an SSH perspective but these troubleshooting steps can also be applied to any connectivity issue.
Keep me posted :)