IBM Support

IBM AIX: Various ssh problems after upgrading to OpenSSH 7.x

Question/Answer


Question

TARGET AUDIENCE:

Users who have upgraded to OpenSSH 7.x from OpenSSH 6.x or earlier

OBJECTIVE:

Provide instructions to resolve a variety of problems after upgrading OpenSSH

Cause

OpenSSH versions 7.1 and higher have more stringent security settings than previous versions of OpenSSH. These include:

  • Root login using a password is no longer enabled by default
  • Various ciphers, key exchange algorithms, key types, and MACs are not enabled by default
     

Users will not be able to log in if their session request includes any of the now disabled-by-default features.

In addition, there are a number of bugs in OpenSSH 7.1.102.1100 and/or 7.5.102.1100 that are fixed in OpenSSH 7.5.102.1500.  These include, but are not limited to:

IV82042: SSHD HANGS DUE TO A RACE CONDITION INVOLVING PTYS
IV87826: X11 FORWARDING USING OPENSSH FAILS IF $DISPLAY IS UNSET
IV89236: SSHD LEAVES PAG VALUE SET EVEN IF KERBEROS AUTHENTICATION FAILS
IV93325: SSH PRINTS "ERROR" AND SFTP FAILS WITH OPENSSH 7.1 SERVER
IV94291: SCP/SFTP FILE COPY PERFORMANCE DROPS IN OPENSSH 7.1
IV96200: ADDING VALUES WITH "+" TO HOSTKEYALGORITHMS DOESN`T WORK
IV97965: THE DEFAULT CIPHER LIST OF SSHD IS INCORRECT
IJ01208: MAN PAGES BROKEN IN OPENSSH 7.5
IJ02099: ALLOWPKCS12KEYSTOREAUTOOPEN FAILS IN OPENSSH 7 SERVER
IJ03122: OPENSSH 7.5 DOES NOT CREATE USER`S HOME DIRECTORY
IJ03680: COMPRESSION BROKEN IN OPENSSH 7.5
IJ05383: SSHD CRASH WITH KERBEROS

There is also an undocumented change starting in OpenSSH 7.5.102.1100 that reverses an errant change in 7.1.102.1100 which prevented sshd from exiting after prompting for a password change if UseLogin was set to false (the default).  The intent of the open source community is that sshd should always exit after a user changes their password during the authentication process (for example, due to the password being expired).  OpenSSH 7.5.102.1100 restores the functionality intended by the open source community.

Although not specific to OpenSSH, there are known bugs in AIX which may cause logins to hang.  Note that IJ04641 and IJ06228 only apply to multi-threaded products such as Tectia.

IV96177: LOGINS HANG TRYING TO WRITE TO UTMP
IJ04641: _ATFORK_PREPARE HANG
IJ06228: _ATFORK_PREPARE HANG WITHOUT LDAP

Answer

Short Answers:

To re-enable DSA keys, add the following lines to /etc/ssh/sshd_config on the server side and/or just the line PubkeyAcceptedKeyTypes to /etc/ssh/ssh_config on the client side.

HostKeyAlgorithms +ssh-dss
PubkeyAcceptedKeyTypes +ssh-dss


Note:  Because these settings add ssh-dss to the end of the respective options, this may not resolve the problem on the ssh client side if there are multiple key types in users' known_hosts file for the server.  In this case, set HostKeyAlgorithms in /etc/ssh/ssh_config on the client to the full list of host key types with ssh-dss at the beginning.

HostKeyAlgorithms ssh-dss,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ssh-ed25519,ssh-rsa


To re-enable the old Diffie-Hellman KEX (key exchange) algorithm, add the following line to /etc/ssh/sshd_config and/or /etc/ssh/ssh_config.

KexAlgorithms +diffie-hellman-group1-sha1


To enable the same ciphers as in OpenSSH 6.x (plus the new ciphers available in OpenSSH 7.x), add the following line to /etc/ssh/sshd_config and/or ssh_config.

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com


To re-enable root logins with a password, change the PermitRootLogin option in /etc/ssh/sshd_config to this.

PermitRootLogin yes


The following error means that the other side does not support Diffie-Hellman keys based on moduli of at least 2048 bits. The only options are to continue using OpenSSH 6.x on AIX or upgrade the software on the other side to a level that supports moduli of at least 2048 bits.

ssh_dispatch_run_fatal: Connection to X.X.X.X port 22: DH GEX group out of range


Long Answers:

Root Login

By default in OpenSSH 7.1 and higher, the root user may only log in using non-interactive methods, such as with a key. To allow root to log in using a password, change the PermitRootLogin setting in /etc/ssh/sshd_config to this:

PermitRootLogin yes

Ciphers

In the AIX versions of OpenSSH prior to 7.1, the default cipher list was the same as the list of allowed ciphers:

aes128-ctr
aes192-ctr
aes256-ctr
arcfour256
arcfour128
aes128-cbc
3des-cbc
blowfish-cbc
cast128-cbc
aes192-cbc
aes256-cbc
arcfour
rijndael-cbc@lysator.liu.se

However, in OpenSSH 7.1.102.1100, the default allowed cipher list was shortened to just these:

aes128-ctr
aes192-ctr
aes256-ctr
chacha20-poly1305@openssh.com,
aes128-gcm@openssh.com
aes256-gcm@openssh.com

The other ciphers are still present in ssh, but they are not allowed by default. If a client tries to connect using one of the ciphers not on the allowed cipher list and the client won't one of the ciphers on the allowed cipher list, the connection attempt will be rejected by the server with the error message "no matching cipher found."

To allow specific or additional ciphers in the sshd server, use the "Ciphers" option in /etc/ssh/sshd_config. You can specify a list of allowed ciphers or add individual ciphers with the "+" option. Examples:  

Ciphers blowfish-cbc,aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com

Ciphers +blowfish-cbc,arcfour256,arcfour128


To allow specific or additional ciphers in the ssh client, use the same Ciphers option but instead set it in either the global /etc/ssh/ssh_config file or each user's individual $HOME/.ssh/config file.

Notes:

  • The "Ciphers" option is a single line.
  • The ciphers in the "Ciphers" option are separated by commas without spaces.
  • Ciphers have a higher preference the earlier they appear in the list.
  • There is a bug in OpenSSH 7.1 (7.1.102.1100) that prevents the "+" option from working. This is fixed in OpenSSH 7.5 (7.5.102.1100).
     

Key Exchange Algorithms

In OpenSSH 7.1 and higher, the Diffie-Hellman Group 1 SHA1 algorithm is no longer allowed by default.  If the client does not support other key exchange algorithms, the connection may fail with the message "no matching key exchange method found."

To allow specific or additional key exchange algorithms in the sshd server, use the "KexAlgorithms" option in /etc/ssh/sshd_config. You can specify a list of allowed key exchange algorithms or add individual algorithms with the "+" option. Examples:

KexAlgorithms diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,curve25519-sha256@libssh.org,gss-group1-sha1-,gss-group14-sha1-,gss-gex-sha1-

KexAlgorithms +diffie-hellman-group1-sha1


To allow specific or additional key exchange algorithms in the ssh client, use the same KexAlgorithms option but instead set it in either the global /etc/ssh/ssh_config file or each user's individual $HOME/.ssh/config file.

Notes:

  • The "KexAlgorithms" option is a single line.
  • The algorithms in the "KexAlgorithms" option are separated by commas without spaces.
  • Kex algorithms have a higher preference the earlier they appear in the list.
     

DSA Keys

In OpenSSH 7.1 and higher, DSA key types are no longer allowed by default.  There are two separate settings which control the use of key types.

HostKeyAlgorithms specifies which key types algorithms may be offered by the server or accepted by the client. Even if the server has a DSA key file and that key file is referenced by the HostKey option, it will not be used unless the "ssh_dss" algorithm is present in the HostKeyAlgorithms list.

To allow specific or additional key type algorithms in the sshd server, use the "HostKeyAlgorithms" option in /etc/ssh/sshd_config. You can specify a list of allowed key type algorithms or add individual algorithms with the "+" option. Examples:

HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ssh-ed25519,ssh-rsa,ssh-dss

HostKeyAlgorithms +ssh-dss


To have the ssh client accept specific or additional key exchange algorithms, use the same HostKeyAlgorithms option but instead set it in either the global /etc/ssh/ssh_config file or each user's individual $HOME/.ssh/config file.

PubkeyAcceptedKeyTypes specifies which public key types are accepted by the server or offered by the client.

To allow specific or additional public key types from an ssh client, use the "PubkeyAcceptedKeyTypes" option in /etc/ssh/sshd_config on the sshd server. You can specify a list of allowed key types or add individual types with the "+" option. Examples:

PubkeyAcceptedKeyTypes ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss

PubkeyAcceptedKeyTypes +ssh-dss


To have the ssh client offer specific or additional key types, use the same HostKeyAlgorithms option but instead set it in either the global /etc/ssh/ssh_config file or each user's individual $HOME/.ssh/config file.

Notes:

  • The "HostKeyAlgorithms" and "PubkeyAcceptedKeyTypes" options are single lines.
  • The values in both options are separated by commas and without spaces.
  • Values have a higher preference the earlier they appear in the list.
     

MAC Algorithms

Certain MAC algorithms, most notably hmac-md5 and hmac-md5-96, are no longer allowed by default. If the client does not support newer MAC algorithms, the connection may fail with the message "no matching MAC found."

To allow specific or additional MAC algorithms in the sshd server, use the "Macs" option in /etc/ssh/sshd_config. You can specify a list of allowed MAC algorithms or add individual algorithms with the "+" option. Examples:

Macs umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-md5-96

Macs +hmac-md5,hmac-md5-96


To have the ssh client use additional MAC algorithms, use the same MAC option but instead set it in either the global /etc/ssh/ssh_config file or each user's individual $HOME/.ssh/config file.

PrintLastLog Option

Between the 6.x and 7.x versions of OpenSSH, the open source community addressed a bug where compiling OpenSSH with DISABLE_LASTLOG did not mark the PrintLastLog option as being unsupported. Since this is now fixed, if the option PrintLastLog appears in the sshd_config file, sshd may display an error message stating that PrintLastLog is an unsupported option. This error is not fatal.

AuthorizedKeysFile Option

While the internal default setting for the AuthorizedKeysFile option has not changed, the entry for it in /etc/ssh/sshd_config is now uncommented by default. This overrides the internal setting in sshd.

When this option is commented out or not present, sshd will search the user's $HOME/.ssh/authorized_keys and $HOME/.ssh/authorized_keys2 files. With the new sshd_config file where AuthorizedKeysFile is uncommented, sshd will only search the $HOME/.ssh/authorized_keys file.

Note that when upgrading from a previous version of OpenSSH, the sshd_config file is not overwritten. Therefore, upgrading to OpenSSH 7.x where the previous version had the AuthorizedKeysFile option commented out will not cause a behavior difference in searching for matching keys. Only new installations or those where OpenSSH was removed entirely before installing a 7.x version will see this behavior difference.

To restore the original behavior, comment out the AuthorizedKeysFile entry in /etc/ssh/sshd_config.

UseDNS Option

Between the 6.x and 7.x versions of OpenSSH, the default value for the UseDNS option changed from "yes" to "no".  With this change, sshd no longer converts a client's IP address back into a host name.

This change causes host-based authentication to fail.  It also prevents sshd from putting the client system's host name in /etc/utmp, which is used by commands like "who" to display a list of logged-in users.  Instead, it writes the client's IP address to /etc/utmp.

To restore the 6.x behavior, add the following option to /etc/ssh/sshd_config:

UseDNS yes


"DH GEX group out of range" Error

Ephemeral keys used by the Diffie-Hellman algorithm are based on moduli (pre-generated prime numbers) of various sizes.  As of OpenSSH 7.x, the minimum bit size of a modulus has increased to 2048.  Very old SSH implementations may not support moduli this large.  In this case, OpenSSH 7.x will display an error similar to this.

ssh_dispatch_run_fatal: Connection to X.X.X.X port 22: DH GEX group out of range


The minimum and maximum moduli sizes are hard-coded in OpenSSH 7.x and cannot be changed through a configuration option.  If the SSH software on the opposite end does not support moduli of at least 2048 bits, there are only two options to resolve the problem:  1) Continue using OpenSSH 6.x on AIX, or 2) upgrade the software on the other side to a version that supports moduli of at least 2048 bits in length.

SUPPORT:

If additional assistance is required after completing all of the instructions provided in this document, please follow the step-by-step instructions below to contact IBM to open a case for software under warranty or with an active and valid support contract.  The technical support specialist assigned to your case will confirm that you have completed these steps.

a.  Document and/or take screen shots of all symptoms, errors, and/or messages that might have occurred

b.  Capture any logs or data relevant to the situation.

c.  Contact IBM to open a case:

   -For electronic support, please visit the IBM Support Community:
     https://www.ibm.com/mysupport
   -If you require telephone support, please visit the web page:
      https://www.ibm.com/planetwide/

d.  Provide a good description of your issue and reference this technote

e.  Upload all of the details and data to your case

   -You can attach files to your case in the IBM Support Community
   -Or Upload data to IBM testcase server analysis:

    http://www.ibm.com/support/docview.wss?uid=ibm10733581

f.  Click here to submit feedback for this document.


Related information

OpenSSH Legacy Options
sshd_config
ssh_config

Document information

More support for: AIX Enterprise Edition

Component: --

Software version: Version Independent

Operating system(s): AIX

Reference #: T1025737

Modified date: 20 June 2019