I hit the same problem. In my case the problem was related to a downloaded JDK that included source code for Java development. This new JDK was installed at the default location /Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/. Some of my scripts use this version of Java and those are the ones that were giving the warning. I was not able to add this Java to firewall settings because the stupid UI would not allow me to navigate to that directory. I am not sure what prevented it from going past /Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/ directory but that was the reason why I could not add an allow rule for that java executable. Anyways, the command line solution to this was to run -
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /Library/Java/JavaVirtualMachines/1.6.0_26-b03-383.jdk/Contents/Home/bin/java
This added the second java into firewall rules. In your case I would suspect there is another version of java executable that is run to the one that you have already allowed. If you right click and the java rule and do show in finder you will probably be able to figure out if you are indeed using the allowed java.
I also have the same problem with a Java application. Every time I open the application a popup appears asking me if I want to allow incoming connections.
The thing is, I already allowed this appliaction in the firewall list. Tried to delete it and add it again and the problem persists.
I'm using a mac application (.app) that is java coded. It's not even a jar file, so I don't understand.
This is really annoying, hope it gets fixed soon.