SSH Timeouts on 64-bit Mageia 3

[linuxdad]

Before you say anything, I have already RTFM'd the Google DB, and I also used the settings that I have working already for 32-bit Mageia 3.

My issue is that the 64-bit Magiea ssh connections continue to timeout even thought the exact same configuration settings are being used in the 32-bit distribution.

My sshd_config files include:

ClientAliveInterval 60
TCPKeepAlive yes

I cannot identify any other settings, unless there is something with msec interfering (or closing) a persistent connection.

Thank you.

[doktor5000]

Can you at least mention ssh connections from which source to what destination?

[wintpe]

i take it that you are talking about ssh'ing into mageia.

we had a similar load of issues when first rolling out redhat here where i work.

but you have two of the settings we used already.

the other we used i dont think will make a difference, in this case.

ClientAliveCountMax 3
MaxStartups 100

however must say i only use mageia as a workstation never a server.

and if you are doing the same, then obviously has to be the destination that counts.

another cause of timeouts is of course in route firewalls and switches, that may time out the connections

ive seen cisco nexus switch settings also cause timeouts, if aggressive settings are in place.

so it would be helpful for anyone offering help if you can describe the source and destination

and what route it takes.

is it a simple layer 3 switch between two hosts , or is it more complex.

it could be related to the high number of network packet drops that your are suffering

regards peter

[jiml8]

While I don't recall all the details, I do recall that there was a recent change in sshd_config that affected connectivity; I had to debug this myself when I migrated to Mageia, bringing over a configuration from a 32 bit Mandriva.

One change I encountered covered root logins; I now have mine set to PermitRootLogin-without-password (meaning that shared key logins are permitted). This also requires PubKeyAuthentication yes, of course.

I also have mine set for UsePAM yes (and I think this is Mageia3 default, though I am not sure about Mandriva...).

You might also look at the AllowUsers setting; if it is enabled only the whitelist of users can log in.

In any case, sshd works fine here, though as I say I did have to tweak it a bit to make it so.

Also, of course, make sure your firewall is leaving the ssh port open.

[wintpe]

thats

PermitRootLogin without-password

been using that with certificate authentication since June 2000

not a new setting

to use this feature you needs to generate keys

ie ssh-keygen -t rsa

and copy the pub key to targets authorised_keys file

unless hes using this which i dont think he is, its not related

he could try starting the server on another port in debug mode

debug mode is one shot, so it will die when the connection test is finished.

on target server start ssh as follows.

sshd -ddd -p 666

this will create quite some output

now start client on source system

ssh -p 666 target

now wait till system times out

you may get an indication of wether sshd is timing out or if something interupted it from the
debug output

another test he could do is setup a ttcp server and client and test the performance of the connection using udp and tcp
this is how we came to the concludtion that broadcome cards with offload switched oncan effect network errors and performace.

ttcp uses a stream of characters from chargen to create a bulk of network transfer either over udp or TCP

we found while udp was fine the tcpoffload was causing timeouts and errors on tcp connections which ssh is..

regards peter

[linuxdad]

Ok, I am still trying to work through the issues with the 64 bit version of sshd disconnecting (mind you this is not occurring on me in the 32-bit mode).

So, running the server as prescribed above, I see the following as output:

Code: Select all


debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1
debug1: Received SIGCHLD.
debug1: session_by_pid: pid 28024
debug1: session_exit_message: session 0 channel 0 pid 28024
debug2: channel 0: request exit-status confirm 0
debug1: session_exit_message: release channel 0
debug2: channel 0: write failed
debug2: channel 0: close_write
debug2: channel 0: send eow
debug2: channel 0: output open -> closed
debug1: session_pty_cleanup: session 0 release /dev/pts/3
debug2: channel 0: read<=0 rfd 10 len -1
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel 0: send close
debug2: notify_done: reading
debug3: channel 0: will not send data after close



What is the significance of the 19 server_input_global_request: lines? I believe these occur every 30 seconds, which would mean that there was a 10 minute timer for something.

The only thing that I can detect is the following sshd_config entry:

#MaxStartups 10:30:100

(although the 32-bit version has this in the configuration file, and it does not behave similarly).

Does anyone have a clue as to what is going on?

[doktor5000]

Ok, I am still trying to work through the issues with the 64 bit version of sshd disconnecting (mind you this is not occurring on me in the 32-bit mode).

So, running the server as prescribed above, I see the following as output:

Code: Select all

debug1: server_input_global_request: rtype keepalive@openssh.com want_reply 1



What is the significance of the 19 server_input_global_request: lines? I believe these occur every 30 seconds, which would mean that there was a 10 minute timer for something.

Those are keepalive packages. See

ServerAliveCountMax
Sets the number of server alive messages (see below) which may be sent without ssh(1) receiving any messages back from the server. If
this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is
important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent
through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The
server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive.

The default value is 3. If, for example, ServerAliveInterval (see below) is set to 15 and ServerAliveCountMax is left at the default, if
the server becomes unresponsive, ssh will disconnect after approximately 45 seconds. This option applies to protocol version 2 only.

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the
encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.
This option applies to protocol version 2 only.

and related:

TCPKeepAlive
Specifies whether the system should send TCP keepalive messages to the other side. If they are sent, death of the connection or crash of
one of the machines will be properly noticed. However, this means that connections will die if the route is down temporarily, and some
people find it annoying.

The default is “yes” (to send TCP keepalive messages), and the client will notice if the network goes down or the remote host dies. This
is important in scripts, and many users want it too.

To disable TCP keepalive messages, the value should be set to “no”.

man sshd_config has similar options to configure this on the server side.

Do you have anything nondefault in ~/.ssh/config or /etc/ssh/ssh_config on the client side or in /etc/ssh/sshd_config on the server side?

Those should be the Mageia defaults:

Code: Select all

┌─[doktor5000@Mageia4]─[19:57:57]─[~]
└──╼ sudo grep -i alive /etc/ssh/*config
/etc/ssh/sshd_config:#TCPKeepAlive yes
/etc/ssh/sshd_config:#ClientAliveInterval 0
/etc/ssh/sshd_config:#ClientAliveCountMax 3

[doktor5000]

Some more context information: http://linux.derkeiler.com/Mailing-List ... /1826.html

[linuxdad]

I have the following settings for Keep Alive, and I have been testing different values on the Server and the Client settings. I want to repeat that I am experienced with SSH and have been using it for many years, but I am still baffled as to why this is not functioning as anticipated.

Code: Select all

grep -i alive /etc/ssh/*config
/etc/ssh/ssh_config:    ServerAliveInterval 30
/etc/ssh/sshd_config:TCPKeepAlive yes
/etc/ssh/sshd_config:#ClientAliveInterval 60
/etc/ssh/sshd_config:ClientAliveInterval 0
/etc/ssh/sshd_config:ClientAliveCountMax 99999


[linuxdad]

Still looking for answers. So I reviewed the installed RPMs for the 64-bit system:

Code: Select all

 rpm -qa | grep ssh
openssh-6.1p1-4.mga3
openssh-server-6.1p1-4.3.mga3
openssh-clients-6.1p1-4.mga3
openssh-clients-6.1p1-4.3.mga3
openssh-server-6.1p1-4.mga3
lib64ssh2_1-1.4.3-2.mga3
lib64ssh4-0.5.4-1.1.mga3
openssh-6.1p1-4.3.mga3


Now on the 32-bit system:

Code: Select all

rpm -qa | grep ssh
openssh-clients-6.1p1-4.mga3
openssh-6.1p1-4.3.mga3
libssh2-devel-1.4.3-2.mga3
sshd-monitor-0.3-9.mga3
openssh-server-6.1p1-4.mga3
libssh2_1-1.4.3-2.mga3

So from the surface, it would appear that the 64-bit server has multiple instances of openssh (openssh-6.1p1-4.mga3 & openssh-6.1p1-4.3.mga3), openssh-clients (openssh-clients-6.1p1-4.mga3 & openssh-clients-6.1p1-4.3.mga3), and oppenssh-server (openssh-server-6.1p1-4.mga3 & openssh-6.1p1-4.3.mga3).

I am certainly unaware as to why this would/could occur, but seeing at it has, is there a simple manner to clean this mess up while I am remoted into the server (via ssh?).

The sooner this is resolved, the better. This I guess that I will need to validate the other packages installed to verify that nothing has occurred with them as well.

[doktor5000]

Code: Select all

openssh-6.1p1-4.mga3
openssh-server-6.1p1-4.3.mga3
openssh-clients-6.1p1-4.mga3
openssh-clients-6.1p1-4.3.mga3
openssh-server-6.1p1-4.mga3
openssh-6.1p1-4.3.mga3


So from the surface, it would appear that the 64-bit server has multiple instances of openssh (openssh-6.1p1-4.mga3 & openssh-6.1p1-4.3.mga3), openssh-clients (openssh-clients-6.1p1-4.mga3 & openssh-clients-6.1p1-4.3.mga3), and oppenssh-server (openssh-server-6.1p1-4.mga3 & openssh-6.1p1-4.3.mga3).

That installation seems pretty borked. OR you installed i586 and x86_64 packages together.
Check both

Code: Select all

rpm -qa --qf "%{NAME}-%{VERSION}-%{ARCH}\n" | grep openssh
rpm -V openssh-6.1p1-4.mga3 openssh-6.1p1-4.3.mga3 openssh-clients-6.1p1-4.mga3 openssh-clients-6.1p1-4.3.mga3 openssh-server-6.1p1-4.mga3 openssh-server-6.1p1-4.3.mga3

Easiest way to resolve is probably

Code: Select all

urpme -a openssh; urpmi openssh-server openssh-clients

[linuxdad]

Thanks Doc!

I like your Diagnosis, Borked. I think that about sums it up! I will keep you posted.

Thanks.

[doktor5000]

On a related note, you should think about rebuilding your rpm database, as having two versions of the _same_ package is not possible.
See http://www.oldrpm.org/hintskinks/repairdb/

[linuxdad]

Well, Borked it is! Updating you on the status of the RPMS:

Code: Select all

rpm -qa --qf "%{NAME}-%{VERSION}-%{ARCH}\n" | grep openssh
openssh-6.1p1-x86_64
openssh-server-6.1p1-x86_64
openssh-clients-6.1p1-x86_64
openssh-clients-6.1p1-x86_64
openssh-server-6.1p1-x86_64
openssh-6.1p1-x86_64


Code: Select all

rpm -V openssh-6.1p1-4.mga3 openssh-6.1p1-4.3.mga3 openssh-clients-6.1p1-4.mga3 openssh-clients-6.1p1-4.3.mga3 openssh-server-6.1p1-4.mga3 openssh-server-6.1p1-4.3.mga3
S.5......    /usr/bin/ssh-keygen
S.5......    /usr/bin/ssh-keyscan
S.5......    /usr/lib64/ssh/ssh-keysign
..5......    /usr/lib64/ssh/ssh-pkcs11-helper
S.5......  c /etc/ssh/ssh_config
..5......    /usr/bin/scp
..5......    /usr/bin/sftp
.M5...G..    /usr/bin/ssh
..5......    /usr/bin/ssh-add
S.5......    /usr/bin/ssh-agent
S.5......  c /etc/ssh/ssh_config
.M....G..    /usr/bin/ssh
SM5...G..  c /etc/ssh/sshd_config
..5......    /usr/lib64/ssh/sftp-server
..5......    /usr/sbin/sshd
SM5...G..  c /etc/ssh/sshd_config


I'm going to check into rebuilding the RPM DB first.