SSH running but not responding

Pete · April 8, 2021, 12:52pm

This may well not be Hestia CP related, so feel free to point me elsewhere.

A bit of background: I had a poorly written PHP script crash the server (ate up way too much memory then died), and no response from anything on it, so my ISP did an ugly power recycle to get it back up. Hestia sprang back into life and the server status screen had everything as running, including SSH but that wasn’t responding.

$ ssh -v [email protected]
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/pete/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to x.x.x.x [x.x.x.x] port 22.
debug1: connect to address x.x.x.x port 22: Operation timed out
ssh: connect to host x.x.x.x port 22: Operation timed out

Clicking Restart on SSH in Hestia appears to have no effect - it says it’s up for the same time it was before clicking Restart, and pressing Stop doesn’t stop it.

I restarted the whole server by pressing the big scary red Restart button, and after a pensive wait the server came back up and I could get back into Hestia. However, SSH is still doing the same thing, so can’t log in via Terminal or SFTP to the server to have a poke around in the logs.

Any advice as to anything I can do via Hestia or where I should be looking next as a way to get back in?

Server Config:
Intel Xeon E3-1230
32GB RAM
2x 2TB SATA HDD 7200rpm 3.5"
Ubuntu 18.04 LTS 64bit
Hestia v1.3.5

top - 13:51:18 up 35 min,  0 users,  load average: 0.02, 0.15, 0.31
Tasks: 192 total,   1 running, 126 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.4 sy,  0.0 ni, 94.6 id,  4.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32929984 total, 28798748 free,  2755928 used,  1375308 buff/cache
KiB Swap:   999420 total,   999420 free,        0 used. 29776060 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 5722 root      20   0   44384   3936   3324 R  12.5  0.0   0:00.02 top
	1 root      20   0  225472   9172   6752 S   0.0  0.0   0:09.78 systemd
	2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
	4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/0:+
	6 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 mm_percpu_+
	7 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/0
	8 root      20   0       0      0      0 I   0.0  0.0   0:01.49 rcu_sched
	9 root      20   0       0      0      0 I   0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.02 migration/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/0
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/0
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/1
   14 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/1
   15 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 migration/1
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/1
   18 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/1:+
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/2
   20 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/2
   21 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 migration/2
   22 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/2
   24 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/2:+
   25 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/3
   26 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/3
   27 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 migration/3
   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/3
   30 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/3:+
   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/4
   32 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 watchdog/4
   33 root      rt   0       0      0      0 S   0.0  0.0   0:00.06 migration/4
   34 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/4

Ta

Pete

falzo · April 8, 2021, 1:48pm

while you posted the top output, that’s rather not interesting because nothing going on. the process tree that’s printed below in Hestia would be more of interest

there are mutliple things that could be in play here: ssh did not start and Hestia detects it’s wrong for whatever reason, the ssh port is blocked (or not been opened) by iptables or ssh could not bind to it (on the public IP at least), or the IP you are trying to connect from has been blocked by fail2ban or another firewall is in play etc.

the best way would be to use some kind of VNC console or IPMI to connect to the server without ssh.
if that’s not available I’d recommend to start the server in rescue mode, mount you disk and check logfiles about what is going on.

Pete · April 8, 2021, 2:29pm

Thanks falzo,

I did check the fail2ban tables as my first port of call, but my IP is nowhere to be seen. I guess this is what you mean by the process tree?:

systemd-+-accounts-daemon---2*[{accounts-daemon}]
		|-agetty
		|-apache2-+-apache2
		|         `-2*[apache2---26*[{apache2}]]
		|-clamd---{clamd}
		|-cron---cron---sh---run-parts---xenmonitor---perl
		|-dbus-daemon
		|-dovecot-+-anvil
		|         |-auth
		|         |-config
		|         |-8*[imap]
		|         |-8*[imap-login]
		|         |-log
		|         `-ssl-params
		|-exim4---exim4
		|-fail2ban-server---12*[{fail2ban-server}]
		|-freshclam
		|-hestia-nginx---hestia-nginx
		|-hestia-php-+-3*[hestia-php]
		|            `-hestia-php---sh---sudo---v-list-sys-cpu----pstree
		|-irqbalance---{irqbalance}
		|-lvmetad
		|-mariadbd---19*[{mariadbd}]
		|-named---10*[{named}]
		|-networkd-dispat---{networkd-dispat}
		|-nginx---9*[nginx]
		|-php-fpm7.4
		|-redis-server---3*[{redis-server}]
		|-rsyslogd---3*[{rsyslogd}]
		|-spamd---2*[spamd child]
		|-sshd
		|-systemd-journal
		|-systemd-logind
		|-systemd-network
		|-systemd-resolve
		|-systemd-timesyn---{systemd-timesyn}
		|-systemd-udevd
		|-toolsclient---21*[{toolsclient}]
		|-unattended-upgr---{unattended-upgr}
		|-vsftpd
		`-xinetd

I can ask my ISP to hook up a KVMoIP to the server, so I can poke around remotely, but what would I be looking for? I appreciate this is now straying far from the Hestia CP path, so even a Google search term so I can learn what to do would be welcome.

Cheers,

Pete

Pete · April 8, 2021, 3:49pm

Still waiting for my ISP to hook up a KVMoIP so I can take a look at the server directly, but in the meantime I’ve been Googling away and here’s what I know so far;

sshd is running, according to Hestia’s GUI and the process tree.
Port 22 is set to allow for SSH / TCP IP address 0.0.0.0/0
My own IP doesn’t appear in fail2ban block list, and I’ve also tried a couple of VPN connections from different IP addresses which makes no difference.
I have 2 other Hestia boxes, which I can SSH into without a problem, so my local machine/network appears to be ok.
All other services on the server (web, mail, ftp, db, etc) appear to be working as normal. Only SSH (and therefore sftp) aren’t playing ball.

As I can’t get in as root right now, I can’t view the SSH log files where I suspect the problem might be hidden. The above checks, and the server crash that seemed to be a little too coincidental, lead me to believe something may be corrupt. Which leads me to the question, can I just re-install SSH potentially with sudo apt-get --reinstall install openssh-server or would I be causing more problems for myself?

Cheers,

Pete

Wibol · April 8, 2021, 4:17pm

You can try enabling SSH access for the “admin” user by selecting the “bash” option in the “SSH access” form at https://sub.domain.tld:8083/edit/user/?user=admin. If you manage to connect, you can escalate privileges with the help of “su”.

Lupu · April 8, 2021, 4:44pm

Posible usefull debug cmds:

ip --brief --color addr               # Confirm server ip
netstat -tlpn | grep ssh              # Confirm ssh port
iptables -n -L | grep YOUR-IP         # Check your ip is not filtered by the firewall
ipset test blacklist YOUR-IP          # Check your ip is not in any blacklist

Pete · April 8, 2021, 4:54pm

No joy on that, unfortunately. I can change the admin user SSH access to bash, but the server still times out trying to connect.

Lupu · April 8, 2021, 5:20pm

you can also check some info from hestia :8083/list/server/?net

Pete · April 8, 2021, 5:27pm

Well after all that, the ISP tells me they’re experiencing a “problem” which means some of their clients can’t log in via SSH… I wished they’d told me! Best part of a day trying to fix something I have no control over.

Thank you to everyone who chipped in some input. This forum remains one of the friendliest and most helpful I’ve experienced - and Hestia is such a fantastic resource.

Cheers,

Pete

falzo · April 8, 2021, 5:54pm

@Pete thanks for adding the additional infos. from the process tree it can be seen that ssh is running, that’s what I wanted to verify

the timeout definitely suggests it’s a network connection issue. if you would suspect your own server aka iptables to be the blocker you could temporarily add allowance for ports 0-65536 TCP and UDP to override the default DROP policy. that could help if you changed your ssh port but somehow forgot about it

@Wibol suggestion is also good if you had troubles with your root access (lost key or password forgotten) - but as you said yourself are you do not even get there for now.

indeed it sounds like there is something in the way on providers side, maybe some messed up setting in their firewall/anti-ddos appliance or whatever they might have in between. best of luck that they will be able to solve it quickly.

and by the way: you should never be afraid to press that big red restart button. services you run on any server should be reboot proof. while great uptime and such might be a thing, I’d rather accept a few minutes downtime for a reboot every now and then but be sure in case it happens without my doing, everything gets back up properly by itself

system · May 8, 2021, 5:55pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.