Tip: AirPort Extreme guest network with internal DNS for primary
After trying for days to figure this out I was finally able to get a working solution and I now have my APE providing a guest and main network while using my lion server as the Dsn server for the main network.
The setup is a bit of a hack and does require you to have at least two devices with staticly assigned ip information on the main network but it does allow you to serve dhcp for both networks from the server and make some services available to the guest network such as iTunes remote for parties.
1) delete your custom Dns entries from the Internet settings in the APE and set two dhcp reservations for .2 and .3 (in this case my Mac mini server and my airport express)
2) reduce the dhcp range to only have 2 available IPs (10.0.1.2-10.0.1.3) and save settings
3) on a computer connected to the main network install wireshark and begin sniffing for packets. Connect at least one device to your guest network and look for any packets that have an ip from your guest network (usually 172.16.42.x) once you capture one of these packets expand the vlan information. This should list a vlan ID ( in my case this was 1003. I would suspect this is universal but do not know)
4) on your server open network preferences, click the gear at the bottom and click "manage virtual interfaces", add a vlan that matches the vlan ID from above. Click ok and apply your settings. The vlan interface should get an ip in the guest network range from your APE.
* if you are running lion you will need to install server admin tools before proceeding*
5) open server admin and add the dhcp service. Create an entry for your primary network (ex: 10.0.1.x) make the dhcp range one higher than the settings in step 2 ( ie: 10.0.1.4 to 10.0.1.253) assign this to the physical interface. Make sure this entry has your internal DNA servers
6) add another entry for the guest networks ip range (ex: 172.16.42.x) again set it one ip higher than step 2 ( 172.16.42.4 to 172.16.42.253) save and activate both ranges. Assign this range to the vlan interface. Make sure this entry either contains your isps dns servers or another public dns server. Turn on dhcp.
Because you have now assigned the only two addresses in the APEs pool for your primary network to static entries there will not be any addresses to assign and the APE will not respond to requests. This will allow your server to pick up the work of assigning IPs. As for your guest network, the APE will assign IPs for two host and then stop. Your clients may either get an IP from the APE or the osx server so both should have the same info. Just make sure the two static clients on your main network have the local DNA servers entered manually.
Mac mini, Mac OS X (10.7.2), Lion Server