pfsense firewall software installs on (old) PC hardware and would sit between your server and Internet, thus not requiring running the OS X server as a gw/router.
But having a separate hardware/software firewall between your OS X server and Internet isn't a bad idéa.
Getting it running natively on Apple hardware would be a seriously hard undertaking (don't know if you can get at the source code - there is a "developer edition").
It would also replace OS X.
I did take a look at it running it on a Mac Intel machine installing it on a virtual disk using Parallels (you need more than one ethernet Interface to get it running).
This might be a better bet:
http://www.macupdate.com/info.php/id/15529/firewall-builder ,
(you can get similar functionality using Fink/Fink Commander and Xcode to install it for free - I think).
There are others:
http://www.macupdate.com/search.php?arch=all&keywords=firewaLL&os=macosx
You talk about using a DMZ. Depending on if this would require having a separate subnet/ethernet interface and wether you want to use public IPs on that subnet (in bridged mode - a "drop in" configuration) you might need to use something else as OS X doesn't support bridging interfaces, you'd need to use NAT and/or routing.
This can help where forwarding ports is required:
http://www.macupdate.com/info.php/id/19881/the-natural
