Snow leopard broke my dns

OK, so it looks like the DNS entries were entered manually. They were not received from DHCP. Did you enter these three servers yourself at some point? I'm just trying to understand if this is the normal config being recommended by your company or if you added these on your own.

I think you can fix this completely by removing the 204.108.253.95 address from your list. The problem is that you have a mixture of internal and external servers and DNS in general doesn't handle that very well. DNS assumes that all servers will return the same results, so it doesn't matter which server you ask, but in your case, server #1 and #2 give different results than server #3, so this is why you're seeing random failures.

Yep, the DNS servers are entered manually, as I have a static IP address at work (I'm special 😉 ) so all of my network settings are manual.

It seems unlikely that this is the result of a generic DNS issue stemming from a mix of internal and external servers, because 1) it's also happening on my home MacBook Pro with only one DNS entry right from DHCP (along with many other people in various threads here and elsewhere), and 2) this never happened before the Snow Leopard upgrade on the identical machines on the identical networks.

From what I've seen so far, it looks like a refresh/caching bug of some kind in Snow Leopard's mDNSResponder, since giving that a brain reset seems to temporarily solve the issues.

Don't assume there's just one "DNS issue". Your home issue might be different. Killing mDNSResponder has the side affect of re-ordering the DNS servers to their default ordering, which temporarily fixes the issue. Snow Leopard tries really hard to find a working DNS server. If a DNS server stops responding or returns an error, Snow Leopard will start using one of the other servers in the list as its primary server. If Snow Leopard decides to use your last server in the list, it means that your internal hostname lookups will start failing because the third server responds with "no such name exists". Snow Leopard assumes that all DNS servers will return the same result and it doesn't matter which server it asks. Having multiple DNS servers in the list is mostly for fault tolerance reasons, in case one server temporarily goes down. Having multiple servers in the list that return different results depending on which one you ask will lead to problems.

That leaves me wondering how Leopard managed to get it right, and Snow Leopard didn't.

It doesn't seem to explain why a "dig" to the same set of DNS servers resolves just fine, and a curl or ping or a variety of other apps' (command-line or otherwise) calls to DNS don't resolve properly.

The home issue is identical to the work issue, and many others are reporting the same issue in a variety of environments, so I suspect something else is up.

In the meantime, however, I'll get rid of that last DNS server and see if it helps any.

Because dig uses the /etc/resolv.conf file directly for its configuration and doesn't go through mDNSResponder.

From the dig(1) man page:

Mac OS X NOTICE
The dig command does not use the host name and address resolution or
the DNS query routing mechanisms used by other processes running on Mac
OS X. The results of name or address queries printed by dig may differ
from those found by other processes that use the Mac OS X native name
and address resolution mechanisms. The results of DNS queries may also
differ from queries that use the Mac OS X DNS routing library.

OK, I feel the need to weigh in again; please let me know if I'm off base here.

Let's take a small network I can think of as an example. There is a single server on the LAN, which provides DNS services. Let's say it's a Windows Domain Controller (DC), for the sake of argument. This DC provides DNS resolution for a Windows Active Directory domain, so clients on the LAN can resolve local services and hosts using DNS. The DNS zone for the LAN AD might be ad.internal.

The DC is also set up to forward queries to an ISP's DNS server. This means that in the event that a client queries for www.google.com (which is not in the ad.internal zone for which the DC is authoritative), the DC forwards the query to the ISP and gets it resolved there. The DC then caches this reply and passes the response back to the client. All is good.

In such an environment, it would also be common practice to set the DHCP service to provide the ISP's DNS server as a second DNS server to all clients. That way, if the DC fails, clients can still resolve internet hosts as when they can't contact the first DNS server (the DC), they'll try the second (the ISP). However, the ISP knows nothing about the ad.internal zone, and never will.

This type of setup is common, and works well for all clients that aren't running Snow Leopard. This includes many Linux distributions, OS X 10.4 and 10.5 and all versions of Windows that I have tested. Snow Leopard is the only DNS client that doesn't work properly in this scenario. Several posts here support this.

My experience is that Snow Leopard fails to correctly use the LAN DNS server, even when the LAN server is fine, and so local, LAN- based ( ad.internal) hosts and services are not resolved. Performing a dig against the LAN server for internal and external hosts shows that both resolve fine, even when telnet, http and other access to LAN hosts is failing through name resolution.

This is a bug in Snow Leopard, whichever way you slice it. I do not believe that the notions that all DNS servers should return the same results or that internal and external DNS servers should not be mixed are correct as without discussing the intricacies of DNS, the simple fact is that this type of configuration is employed by many, many networks and has historically worked fine, even with Leopard. Furthermore, they still work fine for Tiger, Leopard, Windows and Linux DNS clients. It's just Snow Leopard that has problems.

There's been some interesting technical discussion around dig, scutil, dscacheutil and so forth, which has been enlightening to a point, but it should be remembered that dig returns good results even when other processes can't resolve a host. My take on all this is therefore that the internal workings of name resolution on Snow Leopard are different from what we've seen before and possibly broken. The fact that dig resolves a host is no indication that the system will resolve it. Running scutil to determine current DNS service order and then using dig against the first server is therefore flawed - I know from experience that dig will return good results even though other services or applications are failing to resolve that host/FQDN. William Kucharski's last post backs this up.

To demonstrate, I have reproduced the erroneous behaviour on a system here and run *scutil --dns* to see what the current DNS configuration is and I get the following as the first stanza of the output:

resolver #1
search domain[0] : ad.internal
nameserver[0] : 192.168.0.10
nameserver[1] : 192.168.0.254
order : 200000

In the above, 192.168.0.10 is the DC (the internal LAN DNS server for ad.internal) and 192.168.0.254 is the firewall which also acts as a caching-only DNS server. If I dig local or internet hosts @192.168.0.10, I get good results. As an example, I might *dig @192.168.0.10 dc.ad.internal* and get the correct response (note that nslookup also works as expected, giving the correct response). However, if I *ping dc.ad.internal* I get "+ping: cannot resolve dc.ad.internal: Unknown host+". The system is therefore not resolving names the way I would expect.

The workaround, until Apple fix or explain this behaviour is to only specify the internal DNS server(s) to DNS clients, so that they will never try a DNS server that can't resolve internal hosts. If I remove 192.168.0.254 as a DNS server so that only 192.168.0.10 is specified, the problem goes away. The problem is that clients will no longer resolve internet hosts if the internal DNS server fails.

Hope that helps! - if anyone works this out, please post back.

Barnski wrote:
To demonstrate, I have reproduced the erroneous behaviour on a system here and run *scutil --dns* to see what the current DNS configuration is and I get the following as the first stanza of the output:

resolver #1
search domain[0] : ad.internal
nameserver[0] : 192.168.0.10
nameserver[1] : 192.168.0.254
order : 200000

In the above, 192.168.0.10 is the DC (the internal LAN DNS server for ad.internal) and 192.168.0.254 is the firewall which also acts as a caching-only DNS server. If I dig local or internet hosts @192.168.0.10, I get good results. As an example, I might *dig @192.168.0.10 dc.ad.internal* and get the correct response (note that nslookup also works as expected, giving the correct response). However, if I *ping dc.ad.internal* I get "+ping: cannot resolve dc.ad.internal: Unknown host+". The system is therefore not resolving names the way I would expect.

The central issue is simply this:

dig(1) (and host(1) and nslookup(1)) all directly use the DNS resolver and as such the DNS server ordering as present in /etc/resolv.conf.

However, ping(8) uses the *internal Mac OS X name resolution system* which uses a "super DNS search client" which uses the results that are listable via scutil --dns to order queries.

So far so good.

The problem is, looking at the man page for resolver(5):

http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/m an5/resolver.5.html

I see nothing that should cause the disparity of operation that you're seeing.

In particular, for nameserver parameters, it is specifically noted:

Up to MAXNS (currently 3) name servers may be listed, one per keyword. If there are multiple servers,
the resolver library queries them in the order listed. The algorithm used is to try a name server, and
if the query times out, try the next, until out of name servers, then repeat trying all the name
servers until a maximum number of retries are made.

The bit of name resolution that differs from most other systems is the way the DNS Super client routes queries.

However, at least according to the man page, there should be no difference in the way it was handled in Leopard and the way it's handled in SL.

So at the very least, one way to cross check what your Mac's view of the world is is to compare the result as directly obtained from the DNS resolver using dig, host or nslookup, and the result obtained from Directory Services via dscacheutil.

For example, in your case you may wish to compare the output of:

$ nslookup dc.ad.internal

with that of:

$ dscacheutil -q host -a name dc.ad.internal

Interesting.......

*nslookup dc.ad.internal* gives:
Server: 192.168.0.10
Address: 192.168.0.10#53

Name: dc.ad.internal
Address: 192.168.0.10

but *dscacheutil -q host -a name dc.ad.internal* gives no response at all. I just get the prompt back.

So, the system uses directory services for name resolution and directory services is doing something with name resolution that is outside the DNS resolver? - it's clearly not correctly querying the first DNS server specified in any case.

Also, regardless of the internal workings, why is this process happy when there is only one DNS server specified? I mean, it certainly seems as if the second DNS server is used in preference to the first if there are two - this smacks of the "super" DNS client routing queries badly (BTW, my /etc/resolver directory is empty).

I also read that man page and in summary it seems to be saying "+don't assume anything about name resolution - it's complicated and things may or may not be configured and may or may not store configuration data in certain places. You can't possibly tell what's going on with name resolution on your Mac at any given time+". Or am I reading it wrong? 😉

It's not quite as bad as all that.

The resolver is supposed to use the hierarchy specified with the configuration specified by the default resolver issue as the fallback.

I'm curious to know what the full output of scutil --dns looks like for you.

The nameservers are supposed to be used in order for default queries, but if you for some reason have an scutil entry that for whatever reason applies to ad.internal I could see that entry being used instead, based on the published parameters in the aforementioned man page.

We've seen this on systems where people actually used .local as a local domain as that conflicts with the entry for "local" that is used for Bonjour lookups.

From my scutil --dns output:

resolver #1
domain : comcast.net.
nameserver[0] : 208.67.222.222
nameserver[1] : 208.67.220.220
order : 200000
resolver #2
domain : local
options : mdns
timeout : 2
order : 300000
[ … ]

Now I've traced my queries using tcpdump and have yet to see a query going to 208.67.220.220, so it appears that at least on my system the ordering is being honored properly, but my servers are set manually, not via DHCP.

A good thought, but sorry, the .local zone name issue is definitely a red herring - we have generally avoided using .local for DNS in any implementation for a long while, as it used to play havoc with bonjour back in the day, as you say (hence the use of .internal). There is definitely no scutil entry for ad.internal.

tcpdump also confirms the notion that DNS queries hit the second DNS server specified in IP configuration; I set up tcpdump on a test network, and pinged an internal host by name. These lines from tcpdump are interesting:

18:28:35.212202 IP mac-pro.internal.60911 > smoothwall.domain: 58248+ A? testhost.internal. (33)
18:28:35.248439 IP smoothwall.domain > mac-pro.internal.60911: 58248 NXDomain 0/1/0 (108)


To explain the test:
- .internal is the DNS zone name for an internal-only zone
- The LAN DNS server that is authoritative for the .internal zone is at 192.168.0.10
- smoothwall is the firewall/DNS caching-only server at 192.168.0.254
- I had started tcpdump and just tried *ping testhost*
- There is no output prior to this from tcpdump that relates to testhost or testhost.internal
- There is no output from tcpdump that relates to 192.168.0.10 +at all+

Under the same conditions, *scutil --dns* gives:

resolver #1
search domain[0] : internal
nameserver[0] : 192.168.0.10
nameserver[1] : 192.168.0.254
order : 200000

So, even though scutil, network prefs and even good old /etc/resolv.conf show that the first DNS server is 192.168.0.10, a ping causes the system to try a lookup to the second DNS server at 192.168.0.254. The first DNS server is never even queried.

Barnski wrote:
Interesting.......

*nslookup dc.ad.internal* gives:
Server: 192.168.0.10
Address: 192.168.0.10#53

Name: dc.ad.internal
Address: 192.168.0.10

but *dscacheutil -q host -a name dc.ad.internal* gives no response at all. I just get the prompt back.

So, the system uses directory services for name resolution and directory services is doing something with name resolution that is outside the DNS resolver? - it's clearly not correctly querying the first DNS server specified in any case.

I'm seeing this same behavior: when this issue hits, dig or nslookup will properly return the IP of a local hostname, but dscacheutil doesn't. I got the same result as Barnski, a completely empty result from dscacheutil.

Furthermore, issuing +dscachutil -flushcache+ and then reissuing the dscacheutil host query again shows a definite delay while it's doing something....but then the same empty result is returned.

Bouncing the mDNSResponder service with +sudo killall mDNSResponder+ cures all....until the next time it happens.

This indicates that mDNSResponder is definitely broken for some local scenarios, no doubt about it. The DNS lookups being done in these situations are an expected, supported and very common scenario that it has worked properly for in the past, but are broken in Snow Leopard. It's a bug that hopefully will be fixed in 10.6.1.

I suggest everyone experiencing this report it to Apple via either the consumer ( http://www.apple.com/feedback/macosx.html) or developer ( http://developer.apple.com/bugReporter/) bug reporting mechanisms.

Barnski wrote:
A good thought, but sorry, the .local zone name issue is definitely a red herring - we have generally avoided using .local for DNS in any implementation for a long while, as it used to play havoc with bonjour back in the day, as you say (hence the use of .internal).

Agreed: I have this same issue with my internal names using my organization's actual domain name, so it's nothing to do with .local or whatever.

Barnski wrote:
So, even though scutil, network prefs and even good old /etc/resolv.conf show that the first DNS server is 192.168.0.10, a ping causes the system to try a lookup to the second DNS server at 192.168.0.254. The first DNS server is never even queried.

Just for fun, can you post the balance of your scutil --dns output?

All of my systems around my house have around six entries.

There's something very strange going on as I haven't been able to do anything to get my machines to hit the second server I have specified.

If I do a:

<pre>tcpdump -n -i interface -vvv -s 0 port domain and dst nameserver2</pre>

I get nothing, no matter how long I let it run.

I assume in your tests 192.168.0.254 is "smoothwall.domain" so something has to be happening to tell the resolver not to use the first nameserver in the list at all.

The only difference is your nameservers are DHCP-specified; mine are manually specified.

Hi Barnski. The system no longer uses Directory Services for DNS caching. It now uses mDNSResponder, so your dscacheutil commands aren't going to work. If you want to look at mDNSResponder's record cache, you'll want to do this...

*sudo killall -INFO mDNSResponder*

Then look in system.log.