OS X Firewall isn't firewalling

The application firewall is not a packet filter and not capable of discriminating between packets from different sources. You need to use pf for that.

No matter how it's configured, the firewall is not, as some imagine, a malware filter. If that's what you expect it to do, forget it. All it will do is bombard you with pointless alerts.

Suppose you enable file sharing, and you allow guest access to certain folders. That means you want people on your local network, but not outsiders, to be able to access those shared folders without having to enter a password. In the default configuration, the firewall will allow that. The router prevents outsiders from accessing the shares, whether the application firewall is on or off. But if your computer is portable and you connect it to an untrusted network such as a public hotspot, the firewall will still allow access to anyone, which is not what you want. It does not protect you in this scenario.

Now suppose you unknowingly install a trojan that steals your data and uploads it to a remote server. The firewall, no matter how it's configured, will not block that outgoing traffic. It does nothing to protect you from that threat.

A more likely scenario: The web browser or the router is compromised by an attacker. The attack redirects all web traffic to a bogus server. The firewall does nothing to protect you from this threat.

Another scenario: You're running a public web server. Your router forwards TCP connection requests on port 80 to your Mac, and the connections are accepted by the built-in web server, which is signed by Apple. The application firewall, still configured as above, allows this to happen. A different attack tries to hijack port 80 and replace the built-in web server. The good news here is that the firewall does protect you; it blocks incoming connections to the malicious server and alerts you. But the bad news is that you've been rooted. The attacker who can do all this can just as easily turn off the firewall, in which case it doesn't protect you after all.

Now suppose you're running a Minecraft server on the local network. It listens on a high-numbered port. You, as administrator, have reconfigured the firewall to pass this traffic. An attacker is able to log in to a standard account on the server. He figures out how to crash Minecraft, or he just waits for you to quit it, and then he binds his own, malicious, Minecraft server to the same port. The firewall blocks his server, and because he's not an administrator, he can't do anything about it. In this scenario, the security is genuine.

Here is a more realistic scenario in which you might have reason to enable the firewall. Your MacBook has sharing services enabled. You want those services to be available to others on a home or office network. When you're on those networks, the firewall should be off. When you move to an untrusted network, you can either turn off all the services, or enable the firewall with a non-default configuration to block them. Blocking is easier: one click instead of several.

I don't know how this service is implemented, but if it's using launchd as a proxy it would not be blocked. In any case, the application firewall is not going to do what you want. You will either get no incoming connections at all, or you'll get connections from anywhere.