You could try removing the exception for SSH and adding it back. Otherwise, I don't know why it doesn't work. A bug, perhaps. As for the correct use of the firewall, see below.
This is a comment on why you might, or might not, want to use the built-in Application Firewall.
The firewall blocks incoming network traffic, regardless of origin, on a per-application basis. By default it's off, and when turned on, it allows applications digitally signed by Apple, and only those applications, to listen on the network. It does not block outgoing traffic, nor can it distinguish between different sources of incoming traffic, nor does it filter traffic by content.
No matter how it's configured, the firewall is not, as some imagine, a malware filter. If that's what you expect it to do, forget it. All it will do is bombard you with pointless alerts.
Consider some scenarios in which you may expect the firewall to be useful.
1. You enable file sharing, and you allow guest access to certain folders. That means you want people on your local network, but not outsiders, to be able to access those shared folders without having to enter a password. In the default configuration, the firewall will allow that to happen. The router prevents outsiders from accessing the shares, whether the application firewall is on or off. But if your computer is portable and you connect it to an untrusted network such as a public hotspot, the firewall will still allow access to anyone, which is not what you want. It does not protect you in this scenario.
2. You unknowingly install a trojan that steals your data and uploads it to a remote server. The firewall, no matter how it's configured, will not block that outgoing traffic. It does nothing to protect you from that threat.
3. A more likely scenario: The web browser or the router is compromised by an attacker. The attack redirects all web traffic to a bogus server. The firewall does not protect you from this threat.
4. You're running a public web server. Your router forwards TCP connection requests on port 80 to your Mac, and the connections are accepted by the built-in web server, which is codesigned by Apple. The application firewall, still configured as above, allows this to happen. An attacker hacks into the system and tries to hijack port 80 and replace the built-in web server with one that he controls. The good news here is that the firewall does protect you; it blocks incoming connections to the malicious server and alerts you. But the bad news is that you've been rooted. The attacker who can do all this can just as easily turn off the firewall, in which case it doesn't protect you after all.
5. You're running a Minecraft server on the local network. It listens on a high-numbered port. You, as administrator, have reconfigured the firewall to pass this traffic. An attacker is able to log in to a standard account on the server. He figures out how to crash Minecraft, or he just waits for you to quit it, and then he binds his own, malicious, Minecraft server to the same port. The firewall blocks his server, and because he's not an administrator, he can't do anything about it. In this scenario, the security is genuine.
6. Here is a more realistic scenario in which you might have reason to enable the firewall. Your MacBook has sharing services enabled. You want those services to be available to others on a home or office network. When you're on those networks, the firewall should be off. When you move to an untrusted network, you can either turn off all the services, or enable the firewall with a non-default configuration to block them. Blocking is easier: one click instead of several.
