-Pierre - Please be sure to read all of Mr. Hoffman's valuable advice. I am going to assume that you have a pretty small environment - with just a few users that actually require access to your OS X Server - and some number of "guest" users that require only access to the internet and nothing else. I'm also assuming that you have very few records in your OS X Server DNS (the required entries for the server itself and possibly for a few local devices). If this is not true - then my suggestions that follow may not be a suitable solution for your issue - but still you may want to consider them - at least as an interim solution.
First: The reason that you are having trouble with the guest network on your Airport router is due to the fact that Airport routers implement the guest network by using a different subnet (typically defaulting to 172.16.42.x/24) for the guest network. The Airport router does not provide any routing between the the two subnets - thus it is not possible for guests to access to 10.0.x.x/24 or vice-versa. The Airport router hands out the same two DNS server address to both the local 10.0.x.x/24 clients as well as guests. Thus - when you specify 10.0.x.x and 8.8.8.8 (regardless of the order) - all clients (including the guest network) receive these two addresses. The DNS resolver (in each client device) - expects that the two DNS server addresses that it receives from DHCP are going to be "logically" reachable (meaning that there is a route to each). The resolver on clients in the 10.0.x.x/24 subnet can reach both 10.0.x.x and 8.8.8.8 - but - the resolver on clients in the guest subnet (172.16.42.x/24) cannot "logically" reach the 10.0.x.x DNS server because it fails with a "no route to host" error - and does not attempt to try the 8.8.8.8 (which is reachable). Resolution "terminates" because one of the two DNS servers cannot logically be reached - even though the server is actually available - and thus there is no access to the internet from the guest network. For clients on the 10.0.x.x/24 subnet - external resolution would still succeed (even if the OS X DNS Server was shutdown) - because both servers are logically reachable - and since 10.0.x.x is down (it is considered as not responding or not available) and the resolver then attempts to query 8.8.8.8 - and clients on the 10.0.x.x/24 subnet would still get internet access to external websites/hosts. The guest network functionality in Airport devices really does NOT work well (or at all) when a local DNS server is involved - and the local DNS server IP is one of the two DNS Server IPs handed out to clients. The Airport router really needs to allow "guest" DNS servers to be specified - but currently that is not an option.
Quick Resolution - without having to change any of your hardware (at least not at this time):
1. On your Airport Router - set your DNS Servers to one of the following:
a. 8.8.8.8 / 8.8.4.4 (Google DNS)
b. 208.67.222.222 / 208.67.220.220
c. Leave the DNS Servers blank (the DNS servers from your ISP will be used - and should show up in a light grey font)
Note: you can enter your search domain here (for example: example.com) The search domain will be handed out to the guest network as well - but it will not really perform any function there - and should not cause any problems either.
The above settings will handout "reachable" DNS servers to every DHCP device on the local as well as the guest network but not provide any access to your local DNS server (this is addressed in the next step).
2. Override DNS server setting only on local client devices that require access to you OS X Server (Desktops, portables, handhelds, etc). This is feasible only in a small environment - but may be worth the effort to get your environment up and running while you determine how to implement a better solution. You do not need to do a manual override on devices such as NAS drives, printers, and other network devices that don't directly access your server.
For Macs: Open system preferences / network - for both Ethernet and Wifi (in the advanced settings) click the DNS tab. In the DNS servers list - remove any servers that are currently listed (the ones that are DHCP provided are grey and will disappear after you enter a manual ip address). Click the + (plus) sign to enter a new server - enter your 10.0.x.x DNS server - and only that server - no other DNS servers should be listed.
For iOS Devices: In the wifi settings connect to each wifi access point that will be used on the network and tap the arrow to the right of the network name. This will bring up the network settings. Tap on the DNS servers (it will be a common separated list of IP addresses - from DHCP) - enter your 10.0.x.x DNS server - and only that server - no other DNS servers should be listed.
For windows PC's and other non-apple devices - use the appropriate procedure for those devices to override the DNS settings and enter your 10.0.x.x DNS server.
Once all the devices needing server access have been overridden you will have accomplished the following:
1. All devices not requiring access to OS X Server - will have internet access - by default from DHCP - Guest network included. If OS X Server is down for maintenance - these devices will still have uninterrupted access to the Internet.
2. Any devices that do require access to OS X Server - will get their IP address and search domain via DHCP - but will use the overridden 10.0.x.x DNS server and will have access to the Internet and to OS X Server services. When OS X Server is down for maintenance - these devices will not have Internet access.
Next Step (for a better solution):
1. Set up profile manager on OS X Server.
2. Enroll the devices (Macs and iOS devices) that require a DNS override.
3. Set up a network profile that you can push to devices that will automatically or manually be pushed to these devices to override the DNS server settings to use OS X Server.
Other Alternatives (requires hardware changes).
Use a different router (ASUS RT-AC68U) - or one that has VLAN style Guest Network functionality (some routers provide multiple guest networks) and do not have the local DNS server conflict.
Attempt to use DHCP service on OS X Server. I don't recommend this if your environment is small. You want to user a built-in router DHCP server whenever possible.
Get an inexpensive Netgear managed switch. This will allow you to set up port-based VLANS. You would need to put a separate Wifi Access point on a VLAN by itself - to provide guest network functionality - but it also may require some redesign of your network. There is planning required for this solution.
The important thing is that you implement a solution that is going to suit your needs and provide security as well as not taking up all of your time. If you are finding that you are spending all your time troubleshooting something or trying to trick something into working correctly - you would be far better off to buy something better to replace it.
Mr. Hoffman - Do you agree?
~Scott