ssh 'Permission denied' after server reconfig

Question

ssh 'Permission denied' after server reconfig

I'm completely at a loss on this.

The history: We did some upgrades and reconfigured a bit on this server (OS/X Server 10.4.10). There were firewall changes and some other service changes, but we never touched the sshd_config as far as I know. Before the reconfig password was disabled, and public keys were required to access the server. Somehow, this was changed during the reconfig, and passwords became allowed and public keys disallowed... we didn't notice right away (for a while I was concerned the server got hacked, but it doesn't seem to be the case).

Now, I've restored the sshd_config and compared it to a known good server. The only thing I've achieved is... now we can't login on the server using public keys or passwords. 😟 I've been over the config many times and reviewed ssh -vvv for hints... and can't figure it out. I'm posting both the sshd_config as well as a connection attempt with ssh -vvv in hopes someone smarter than me can point out the problem.

Finally, all I can add is that from my laptop, I can ssh into the known-good server just fine. Using the same keys, trying to ssh to the bad server fails (see details below).

Thank you!

--- Begin sshd_config: ---

$OpenBSD: sshd_config,v 1.68 2003/12/29 16:39:50 millert Exp $

# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options change a
# default value.

#Port 22
Protocol 2
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# HostKey for protocol version 1
#HostKey /etc/ssh hostkey
# HostKeys for protocol version 2
#HostKey /etc/ssh host_rsakey
#HostKey /etc/ssh host_dsakey

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 768

# Logging
#obsoletes QuietMode and FascistLogging
SyslogFacility AUTHPRIV
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6

RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile .ssh/authorized_keys

# For this to work you will also need host keys in /etc/ssh knownhosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# SACL options
#SACLSupport yes

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to 'yes' to enable PAM authentication (via challenge-response)
# and session processing. Depending on your PAM configuration, this may
# bypass the setting of 'PasswordAuthentication' and 'PermitEmptyPasswords'
UsePAM no

#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no

# no default banner path
#Banner /some/path

# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server

--- Begin ssh -vvv attempt: ---

helios:~ zbeckman$ ssh -vvv ra.local
OpenSSH_4.5p1, OpenSSL 0.9.7l 28 Sep 2006
debug1: Reading configuration data /Users/zbeckman/.ssh/config
debug1: Applying options for ra.local
debug1: Reading configuration data /etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to ra.local [XXXX%en1] port 22.
debug1: Connection established.
debug1: identity file /Users/zbeckman/.ssh/identity type -1
debug1: identity file /Users/zbeckman/.ssh/id_rsa type -1
debug3: Not a RSA1 key file /Users/zbeckman/.ssh/id_dsa.
debug2: key type_fromname: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key type_fromname: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key type_fromname: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key type_fromname: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /Users/zbeckman/.ssh/id_dsa type 2
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.5
debug1: match: OpenSSH_4.5 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.5
debug2: fd 3 setting O_NONBLOCK
debug1: Miscellaneous failure
No credentials cache found

debug1: Miscellaneous failure
No credentials cache found

debug1: SSH2 MSGKEXINIT sent
debug1: SSH2 MSGKEXINIT received
debug2: kex parsekexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie- hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex parsekexinit: ssh-rsa,ssh-dss
debug2: kex parsekexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes1 92-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex parsekexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes1 92-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex parsekexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac- md5-96
debug2: kex parsekexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac- md5-96
debug2: kex parsekexinit: none,zlib@openssh.com,zlib
debug2: kex parsekexinit: none,zlib@openssh.com,zlib
debug2: kex parsekexinit:
debug2: kex parsekexinit:
debug2: kex parsekexinit: first kexfollows 0
debug2: kex parsekexinit: reserved 0
debug2: kex parsekexinit: gss-gex-sha1-toWM5Slw5Ew8Mqkay al2g==,gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-group14-sha1-toWM5Slw5Ew8Mqk ayal2g==,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,d iffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex parsekexinit: ssh-rsa,ssh-dss
debug2: kex parsekexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes1 92-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex parsekexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes1 92-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex parsekexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac- md5-96
debug2: kex parsekexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac- md5-96
debug2: kex parsekexinit: none,zlib@openssh.com
debug2: kex parsekexinit: none,zlib@openssh.com
debug2: kex parsekexinit:
debug2: kex parsekexinit:
debug2: kex parsekexinit: first kexfollows 0
debug2: kex parsekexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2 MSG_KEX_DH_GEXREQUEST(1024<1024<8192) sent
debug1: expecting SSH2 MSG_KEX_DH_GEXGROUP
debug2: dh genkey: priv key bits set: 120/256
debug2: bits set: 482/1024
debug1: SSH2 MSG_KEX_DH_GEXINIT sent
debug1: expecting SSH2 MSG_KEX_DH_GEXREPLY
debug3: check host_inhostfile: filename /Users/zbeckman/.ssh/known_hosts
debug3: check host_inhostfile: match line 15
debug3: check host_inhostfile: filename /Users/zbeckman/.ssh/known_hosts
debug3: check host_inhostfile: match line 18
debug1: Host 'ra.local' is known and matches the RSA host key.
debug1: Found key in /Users/zbeckman/.ssh/known_hosts:15
debug2: bits set: 496/1024
debug1: ssh rsaverify: signature correct
debug2: kex derivekeys
debug2: set_newkeys: mode 1
debug1: SSH2 MSGNEWKEYS sent
debug1: expecting SSH2 MSGNEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2 MSGNEWKEYS received
debug1: SSH2 MSG_SERVICEREQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2 MSG_SERVICEACCEPT received
debug2: key: /Users/zbeckman/.ssh/id_dsa (0x307a00)
debug2: key: /Users/zbeckman/.ssh/identity (0x0)
debug2: key: /Users/zbeckman/.ssh/id_rsa (0x0)
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod isenabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod isenabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Miscellaneous failure
No credentials cache found

debug1: Miscellaneous failure
No credentials cache found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod isenabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/zbeckman/.ssh/id_dsa
debug3: send pubkeytest
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug1: Trying private key: /Users/zbeckman/.ssh/identity
debug3: no such identity: /Users/zbeckman/.ssh/identity
debug1: Trying private key: /Users/zbeckman/.ssh/id_rsa
debug3: no such identity: /Users/zbeckman/.ssh/id_rsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod isenabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug3: userauth_kbdint: disable: no info reqseen
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive).
helios:~ zbeckman$

G5 Powermac, Mac OS X (10.4.10)

Posted on Aug 22, 2007 9:45 PM

Reply

Answer 1

Aug 22, 2007 9:49 PM in response to Zacharias Beckman

Shoot... the forum reformatted my post. Well, anything with a numbered list in front of it is actually commented out. The "# whatever" was converted into "1. whatever" etc.

Reply

Answer 2

davidh

Level 4

1,896 points

Aug 23, 2007 5:08 AM in response to Zacharias Beckman

looks like the (ssh) keys stored on your server got b0rked in whatever else happened.

Reply

Answer 3

Aug 23, 2007 5:19 AM in response to Zacharias Beckman

Hi

Has the self signed certificate got a red or yellow exclamation mark icon by the side of it? Server Admin > fqdn, or IP address or .local name of your server > Settings > Certificates.

Tony

Reply

Answer 4

davidh

Level 4

1,896 points

Aug 23, 2007 8:03 AM in response to Antonio Rocco

Antonio, are you confusing open- ssl with open- ssh which is what the OP's asking about ?

Maybe I missed something but SSL certs should have no part in this.

Reply

Answer 5

Aug 23, 2007 8:15 AM in response to davidh

Hi david

Of course I am!! Doh!! Long day please forgive so sorry, will disappear into the hole I have come from.

Tony

Reply

Answer 6

Aug 24, 2007 11:37 AM in response to Zacharias Beckman

One more footnote: I've been trying to locate any indication in the server logs of what's going wrong, but I can't find anything. My first instinct was to look in /var/log/secure.log but... nothing. Any ideas?

Reply

Answer 7

davidh

Level 4

1,896 points

Aug 24, 2007 1:19 PM in response to Zacharias Beckman

I think the logs samples you've shown already provide a clue.

On the server, replace the /Users/<yourusername>/.ssh/authorized_keys
from a known-good backup. Or recreate it anew.

Reply

Answer 8

Aug 30, 2007 12:46 PM in response to Zacharias Beckman

Hi

You could also try:

cd /Users/usershortname/.ssh

followed by:

sudo rm -R known_hosts

Now try connecting to the host again using ssh. You will be asked to add the RSA key to your known hosts list, type in yes and hopefully that should be it.

Tony

Reply

Answer 9

Aug 31, 2007 8:09 AM in response to davidh

I don't see what you're seeing in my logs. I did recreate the authorized_keys, and reattempted to connect. As before, this is what I'm getting:

debug1: Next authentication method: publickey
debug1: Offering public key: /Users/zbeckman/.ssh/id_dsa
debug3: send pubkeytest
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive
debug1: Trying private key: /Users/zbeckman/.ssh/identity
debug3: no such identity: /Users/zbeckman/.ssh/identity
debug1: Trying private key: /Users/zbeckman/.ssh/id_rsa
debug3: no such identity: /Users/zbeckman/.ssh/id_rsa
debug2: we did not send a packet, disable method

It seems to me that on my end, we send the public key... and then it just quietly moves on to the next protocol. There is no error message or any indication that something went wrong, or why it would move on to the next protocol.

I've done a cksum on my id_dsa.pub and the remote authorized_keys. They are identical. Both files are owned by me, and readable / writable only by me.

Reply

Answer 10

Aug 31, 2007 8:11 AM in response to Antonio Rocco

Tried it. No luck.

And unfortunately, somewhere along the lines our other "known good" server has now ALSO stopped accepting ssh connections from me. Perhaps there is a problem with my laptop's ssh configuration... but I can't figure out what it might be.

Isn't there a server-side log that will show me what's going on and why the connection is failing?

Reply

Answer 11

Aug 31, 2007 8:20 AM in response to Zacharias Beckman

The only other thing I can think of is that a recent OS/X update is the culprit. It would account for why both of our servers are now failing to work with ssh, as they both were recently updated (but not at exactly the same time). Is anyone aware of changes or problems with the latest updates?

Reply

Answer 12

davidh

Level 4

1,896 points

Aug 31, 2007 9:23 AM in response to Zacharias Beckman

I've never seen a successful update break ssh connectivity. I'd suggest you ownload the combo updater for the most recent update (10.4.10) and apply it at both ends.

Reply

Answer 13

Aug 31, 2007 11:09 AM in response to Zacharias Beckman

Hi

Like David I’ve not seen the update break ssh either. The suggestion I made was what I had to do when I applied the update to a test server that I have. The server differs slightly as it is OSX Server on top of OSX Client rather than a clean install.

What about simply disabling Remote login, restarting the server and re-enabling it?

Tony

Reply

Answer 14

Oct 20, 2007 5:33 PM in response to Zacharias Beckman

Update: All of our server systems are now on the latest 10.4.10. Also, none of them allow connections to SSH, unless I turn on password based access again. We don't want to do that.

I'm assuming that 10.4.10 has somehow broken SSH. We certainly didn't make any changes to our configuration across all servers -- but this problem showed up systematically after each machine was upgraded.

If anyone discovers a fix, please post it! In the meantime you might want to avoid 10.4.10 on any servers that require SSH connectivity.

Reply

Answer 15

Jan 7, 2008 8:25 PM in response to Zacharias Beckman

Don't RSA keys.
Use DSA keys.
Don't enable root logins.
Use public/private key login.
Unfortunately you can't use Public/Private Keys if you're running an OD master and any replicas
because the Apple software droogs haven't figured out how to get that to work.
You have to use password auth on the OD master and any replicas, and have root logins
enabled for this to work. Stupid Apple, fix this.
Check the permissions on your id_dsa key files and authorized_keys files.
This line in your log file:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive).

indicates that the perms are wrong on the files in your /users/username/.ssh folder.

fix them so they are like this;

-rw-r----- 1 username groupname 668 Jan 01 2008 authorized_keys2
-rw------- 1 username groupname 668 Jan 01 2008 id_dsa
-rw-r--r-- 1 username groupname 668 Jan 01 2008 id_dsa.pub
-rw-r--r-- 1 username groupname 668 Jan 01 2008 known_hosts

make your /private/etc/ssh_config file like this:

Port 22
Protocol 2
Cipher blowfish
Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
ForwardAgent yes
ForwardX11 no
HostbasedAuthentication no
IdentityFile ~/.ssh/id_dsa
PasswordAuthentication no #Change to yes for OD/replica ssh. Stupid Apple, fix this.
PreferredAuthentications publickey,keyboard-interactive,password
PubkeyAuthentication yes #Change to no for OD/replica ssh. Stupid Apple, fix this.
RhostsAuthentication no
RhostsRSAAuthentication no
RSAAuthentication no

make your /private/etc/sshd_config like this:

Port 22
Protocol 2

HostKey /private/etc/ssh host_dsakey

SyslogFacility AUTH
LogLevel VERBOSE

LoginGraceTime 30
PermitRootLogin no #Change to yes for OD/replica ssh. Stupid Apple, fix this.
StrictModes yes

PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys2

RhostsAuthentication no
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
IgnoreUserKnownHosts yes

PasswordAuthentication no #Change to yes for OD/replica ssh. Stupid Apple, fix this.
PermitEmptyPasswords no

ChallengeResponseAuthentication no

PAMAuthenticationViaKbdInt no

X11Forwarding no
ClientAliveInterval 300
ClientAliveCountMax 6
UsePrivilegeSeparation yes

MaxStartups 5:10:10

Subsystem sftp /usr/libexec/sftp-server

Buy this book:

http://www.oreilly.com/catalog/sshtdg2/

Read. Understand. Apply.

Reply