Yes, needs to be open.
And it's not dangerous, because you only have ntpd listening on port 123 for traffic it initiated, it won't answer ssh connection requests which have to come in through port 22.
This is how ipfw works: It lets specified traffic through on specified ports, not just anything on port 123.