A few weeks ago, our server, running OpenSSH 4.7 on OS X 10.4.11, looked like it was slowing down or even crashing. While the web server was working perfectly, nothing that required a username and password, like ssh or ftp, would allow anyone to log in consistently. Looking at the logs, it turned out that the server was being bombarded with ssh requests, probably hackers looking to exploit the Debian SSH bug. Looking at the logs, the only thing to go on was that each bad ssh login had the message [Sender com.apple.SecurityServer] [PID -1] [Message Failed to authorize right system.login.tty by process /usr/sbin/sshd for authorization created by /usr/sbin/sshd.] in the file /var/log/asl.log

The problem was that every time the attacking machine tried another key/password, it would spawn a new sshd process, which had to communicate with the password services (com.apple.SecurityServer) in order to validate the password. Eventually what ended up happening is that there were so many requests to the password services that they basically ended up just hanging, and anything that required a password: ssh, ftp, etc, just stopped working.

I tried a number of things to get this to work. First, I tried shutting the ssh server off for a short time to see if it was just a single bot and it would move on, but it looks like once the bot knows that port 22 is open, it keeps that IP address in its memory. I upgraded OpenSSH to 5.0, which is the latest version. We tried filtering out the IP addresses, but the attacks were coming from all over the place.

Eventually, it turned out that there was a very simple solution in the OpenSSH config file. The problem was that when the ssh daemon was confronted with an incorrect password, it was checking against the password database multiple times, which was overwhelming the password services. In order to stop this behavior, I ended up changing the line which looks like
#ChallengeResponseAuthentication yes
in the default /etc/sshd_config file to
ChallengeResponseAuthentication no

While this problem does not stop the bots from trying to get bad keys/passwords, it does protect the password services from the attack. We have had no problem logging in to the server since I made that small change in the config file.

Categories: Hackers, OS X, Server Admin