Snow Leopard mini having DNS issues

I have a pretty plain setup involving a mac pro connected via ethernet to an ADSL modem and I am experiencing this issue as well.

$ ping mail.google.com
ping: cannot resolve mail.google.com: Unknown host

$ nslookup mail.google.com
Server: 10.1.1.1
Address: 10.1.1.1#53

Non-authoritative answer:
mail.google.com canonical name = googlemail.l.google.com.
Name: googlemail.l.google.com
Address: 66.249.89.18
Name: googlemail.l.google.com
Address: 66.249.89.83
Name: googlemail.l.google.com
Address: 66.249.89.19
Name: googlemail.l.google.com
Address: 66.249.89.17

$ping mail.google.com
ping: cannot resolve mail.google.com: Unknown host

IPv6 is disabled on the ethernet port

Here is the output of the scutil command

$ scutil --dns
DNS configuration

resolver #1
domain : home.gateway
nameserver[0] : 10.1.1.1
order : 200000

resolver #2
domain : local
options : mdns
timeout : 2
order : 300000

resolver #3
domain : 254.169.in-addr.arpa
options : mdns
timeout : 2
order : 300200

resolver #4
domain : 8.e.f.ip6.arpa
options : mdns
timeout : 2
order : 300400

resolver #5
domain : 9.e.f.ip6.arpa
options : mdns
timeout : 2
order : 300600

resolver #6
domain : a.e.f.ip6.arpa
options : mdns
timeout : 2
order : 300800

resolver #7
domain : b.e.f.ip6.arpa
options : mdns
timeout : 2
order : 301000

Looks like in 10.6 the 1st DNS handling plus caching is no longer done in dscache, but directly in mDNSResponder. Turning on debug logging shows:

1st (successful) try:
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: Adding FD for uid 501
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: DNSServiceCreateConnection START
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: Error socket 44 created 00000000 00000001
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: DNSServiceQueryRecord(www.munnik.net., Addr, 5000) START
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: Error socket 44 closed 00000000 00000001 (0)
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: DNSServiceQueryRecord(www.munnik.net., Addr) ADD 4 www.munnik.net. Addr 192.168.1.121
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: Cancel 00000000 00000001
Aug 31 14:06:14 unused5 mDNSResponder[28]: 43: DNSServiceQueryRecord(www.munnik.net., Addr) STOP
Aug 31 14:06:18 unused5 mDNSResponder[28]: 43: Removing FD

2nd (failed) try:

Aug 31 14:06:26 unused5 mDNSResponder[28]: 43: Adding FD for uid 501
Aug 31 14:06:26 unused5 mDNSResponder[28]: 43: DNSServiceCreateConnection START
Aug 31 14:06:26 unused5 mDNSResponder[28]: 43: Error socket 44 created 00000000 00000001
Aug 31 14:06:26 unused5 mDNSResponder[28]: 43: DNSServiceQueryRecord(www.munnik.net., Addr, 5000) START
Aug 31 14:06:26 unused5 mDNSResponder[28]: 43: Error socket 44 closed 00000000 00000001 (0)
Aug 31 14:06:26 unused5 mDNSResponder[28]: 43: DNSServiceQueryRecord(www.munnik.net., Addr) ADD 0 www.munnik.net. Addr
Aug 31 14:06:26 unused5 mDNSResponder[28]: 43: Cancel 00000000 00000001

Afterwards, the mDNSResponder state log does not show any entries for www.munnik.net

Any suggestions would be greatly appreciated.

This is what I have found:

1. Name resolution comes and goes. I wrote a shell script to do one ping every ten seconds and it shows name resolution failing on and off

Mon 31 Aug 2009 22:10:45 EST
PING googlemail.l.google.com (66.249.89.17): 56 data bytes
64 bytes from 66.249.89.17: icmp_seq=0 ttl=245 time=131.454 ms

--- googlemail.l.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 131.454/131.454/131.454/0.000 ms
Mon 31 Aug 2009 22:10:56 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:11:06 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:11:16 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:11:26 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:11:36 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:11:46 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:11:56 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:12:06 EST
PING googlemail.l.google.com (66.249.89.19): 56 data bytes
64 bytes from 66.249.89.19: icmp_seq=0 ttl=246 time=131.397 ms

--- googlemail.l.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 131.397/131.397/131.397/0.000 ms
Mon 31 Aug 2009 22:12:16 EST
ping: cannot resolve mail.google.com: Unknown host
Mon 31 Aug 2009 22:12:26 EST
ping: cannot resolve mail.google.com: Unknown host

2. If while it is getting unknown host, I flush the cache (i.e. dscacheutil -flushcache), it will immediately be able to ping properly again for a while

3. dscacheutil -cachedump -entries host gives no info (my leopard machine has entries for all recent DNS lookups)

4. When ping gives "unknown host", there is no network activity at all when it gives that result (at least none shown in CocoaPacketAnalyzer)

Coldboot wrote:
3. dscacheutil -cachedump -entries host gives no info (my leopard machine has entries for all recent DNS lookups)

Apparently 10.6 does not uses the directory for caching anymore.
You should check the mDNSResponder cache. You can do this by sending a:
sudo killall -INFO mDNSResponder
Then checking your /var/log/system.log

I have noticed that on my system it contains "empty" entries for the hosts that do not resolve correctly. After a while these entries are cleared out (either by timeout or by flushing) and a single successful lookup can be done (which again results in an empy entry etc.)

Looks like 10.6 somehow does not handle A records received from my DNS server correctly (BIND 9.3.2)

Apparently 10.6 does not uses the directory for caching anymore.
You should check the mDNSResponder cache. You can do this by sending a:
sudo killall -INFO mDNSResponder
Then checking your /var/log/system.log

Thanks for that info. In my mDSNResponder log it goes from:

441 125 -U- Addr 4 googlemail.l.google.com. Addr 66.249.89.83
441 125 -U- Addr 4 googlemail.l.google.com. Addr 66.249.89.19
441 125 -U- Addr 4 googlemail.l.google.com. Addr 66.249.89.18
441 125 -U- Addr 4 googlemail.l.google.com. Addr 66.249.89.17

at one point to

441 45 -U- - Addr 0 googlemail.l.google.com. Addr

when failing.

No idea what causes that transition though.

Hi grmbl99. It looks like you've figured out most of the DNS debugging tricks, but the one thing left is to enable packet logging. That will show you why mDNSResponder ended up creating a negative cache entry for your server. So to recap...

1. Run "sudo killall -USR1 mDNSResponder" to enable operation logging.
2. Run "sudo killall -USR2 mDNSResponder" to enable packet logging.
3. Run "sudo killall -HUP mDNSResponder" to clear the DNS cache.
4. Reproduce the problem.
5. Run "sudo killall -INFO mDNSResponder" to dump mDNSRepsonder's internal state.

Then search through "/var/log/system.log" for "www.munnik.net" and try to find the place where the server responded.

Let us know what you discover.

Looks like I found out what was causing the problem; after browsing the mDNSResponder sourcecode (I love opensource 🙂 )

As far as I understand the mDNSResponder adds a negative entry in the cache because the TTL received from my DNS server is low (was not specified in zone file, so BIND set it to SOA MINTTL).
By explicitly adding a TTL to my zone file the problem seems to be solved.

Why 10.6 adds a negative entry to the cache (without any Adres info), subsequently breaking all gethostbyname's for this entry is unclear to me.

Hope this helps.

@grmbl99

Very well stated. You nailed it. I have this same issue. Exactly the same using all the same command util's. I continue to watch this thread for solution. We happen to be using Win2k3StdEd for our internal DNS server.

Great work.

As for the thousands of the rest of us who are gripped by this oddity & half-broken mostly-broken implementation, what is your advice. We, for example, have a few offices, and most of those run Win2k3 as the internal DNS provider. Others use the firewall which is mostly always pfSense.

I am unclear as to what our best short-term work-around is.

I am unclear what our best long-term work-around is.

Your thoughts are appreciated on this (both short- & long- term).

Cheers.

I've been seeing this on our internal network, where the TTL in our DNS server is 3 hours. What exactly do you mean by a "low value"?

Would setting the TTL to 6 hours fix this?

What fixed the problem for me was adding an explicit TTL to my zone files.
(using freebsd dns server running BIND 9.3.2)

Before:
@ IN SOA ns.grmblbarf.com. hostmaster.grmblbarf.com. (
2009083101
21600 ; refresh after 6 hours
3600 ; retry after 1 hour
604800 ; expire after 1 week
86400 ; minimum TTL of 1 day
)
IN NS ns.grmblbarf.com.
IN MX 10 mail.grmblbarf.com.
localhost A 127.0.0.1
gateway IN A 192.168.1.1
ns IN A 192.168.1.121
mail IN A 192.168.1.121

After:
@ 86400 IN SOA ns.grmblbarf.com. hostmaster.grmblbarf.com. (
2009083101
21600 ; refresh after 6 hours
3600 ; retry after 1 hour
604800 ; expire after 1 week
86400 ; minimum TTL of 1 day
)
IN NS ns.grmblbarf.com.
IN MX 10 mail.grmblbarf.com.
localhost A 127.0.0.1
gateway IN A 192.168.1.1
ns IN A 192.168.1.121
mail IN A 192.168.1.121

Since this fixed my problem, I did not do any further packet dumps on OSX.
If I have some spare time I will do that to see if it shows some more info.

grmbl99, your explanation doesn't fully make sense. mDNSResponder only caches negative answers when the server explicitly returns NXDOMAIN, which means "name does not exist". It would be interesting for you to revert your change and then capture the actual response your server was sending using the USR2 command I sent earlier. That will tell us exactly why it was failing.

Open System Preferences, select "Network"
In the network panel, select your interface (for most this will be either "Ethernet" or "Airport"), then click the "Advanced..." button.

In the sub-panel, click the "DNS" tab in the row across the top.

If your DNS server is listed in bold text (not grey) then it means that somehow you've locally defined your DNS server. You can highlight it and just click the "-" button (left of where it says "IPv4 or IPv6 addresses") and it'll remove the value. This will cause your Mac to revert to whatever value it received via DHCP (which will be shown in grayed-out text and cannot be edited (if you ever want to over-ride that in the future just click the "+" button and add your new DNS server.)

I upgraded 5 macs. Four had no issues. One had this DNS issue but ONLY with the "Ethernet" interface (the "Airport" interface was fine).

Snoop Dogg wrote:
grmbl99, your explanation doesn't fully make sense. mDNSResponder only caches negative answers when the server explicitly returns NXDOMAIN, which means "name does not exist". It would be interesting for you to revert your change and then capture the actual response your server was sending using the USR2 command I sent earlier. That will tell us exactly why it was failing.

Ok it turns out I was a bit too quick too draw my conclusion (I made some other changes to my zone files as well which were the actual solution for me), however there still is something fishy with the TTLS.

The steps described by Snoop Dogg show the following:

1: situation in which error occurs; zone file contains:

@ IN SOA ns.munnik.net. hostmaster.munnik.net. (
2009083103
43200 ; refresh after 12 hrs
3600 ; retry after 1 hour
3600000 ; expire after 42 days
2592000 ; minimum TTL of 30 days

mDNSResponder output:

Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: Adding FD for uid 501
Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: DNSServiceCreateConnection START
Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: Error socket 48 created 00000000 00000001
Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: DNSServiceQueryRecord(mail.munnik.net., Addr, 5000) START
Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: Error socket 48 closed 00000000 00000001 (0)
Aug 31 21:14:09 unused5 mDNSResponder[28]: -- Sent UDP DNS Query (flags 0100) RCODE: NoErr (0) RD ID: 39429 21 bytes from port 55027 to 192.168.1.121:53 --
Aug 31 21:14:09 unused5 mDNSResponder[28]: 1 Questions
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 mail.munnik.net. Addr
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 Answers
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 Authorities
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 Additionals
Aug 31 21:14:09 unused5 mDNSResponder[28]: -- Received UDP DNS Response (flags 8580) RCODE: NoErr (0) AA RD RA ID: 39429 70 bytes from 192.168.1.121:53 to 192.168.1.5:55027 --
Aug 31 21:14:09 unused5 mDNSResponder[28]: 1 Questions
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 mail.munnik.net. Addr
Aug 31 21:14:09 unused5 mDNSResponder[28]: 1 Answers
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 TTL1879048 4 mail.munnik.net. Addr 192.168.1.121
Aug 31 21:14:09 unused5 mDNSResponder[28]: 1 Authorities
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 TTL1879048 15 munnik.net. NS ns.munnik.net.
Aug 31 21:14:09 unused5 mDNSResponder[28]: 1 Additionals
Aug 31 21:14:09 unused5 mDNSResponder[28]: 0 TTL1879048 4 ns.munnik.net. Addr 192.168.1.121
Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: DNSServiceQueryRecord(mail.munnik.net., Addr) ADD 4 mail.munnik.net. Addr 192.168.1.121
Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: Cancel 00000000 00000001
Aug 31 21:14:09 unused5 mDNSResponder[28]: 47: DNSServiceQueryRecord(mail.munnik.net., Addr) STOP
Aug 31 21:14:11 unused5 mDNSResponder[28]: 47: Removing FD



This results in the following entry in the cache:

Aug 31 21:28:44 unused5 mDNSResponder[28]: Slt Q TTL if U Type rdlen
Aug 31 21:14:27 unused5 mDNSResponder[28]: 72 60 -U- - Addr 0 mail.munnik.net. Addr



2: fixed situation; changed zone file contains

@ IN SOA ns.munnik.net. hostmaster.munnik.net. (
2009083104
21600 ; refresh after 6 hours
3600 ; retry after 1 hour
604800 ; expire after 1 week
86400 ; minimum TTL of 1 day

mDNSResponder output:

Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: Adding FD for uid 501
Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: DNSServiceCreateConnection START
Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: Error socket 47 created 00000000 00000001
Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: DNSServiceQueryRecord(mail.munnik.net., Addr, 5000) START
Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: Error socket 47 closed 00000000 00000001 (0)
Aug 31 21:39:36 unused5 mDNSResponder[28]: -- Sent UDP DNS Query (flags 0100) RCODE: NoErr (0) RD ID: 29534 21 bytes from port 63091 to 192.168.1.121:53 --
Aug 31 21:39:36 unused5 mDNSResponder[28]: 1 Questions
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 mail.munnik.net. Addr
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 Answers
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 Authorities
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 Additionals
Aug 31 21:39:36 unused5 mDNSResponder[28]: -- Received UDP DNS Response (flags 8580) RCODE: NoErr (0) AA RD RA ID: 29534 70 bytes from 192.168.1.121:53 to 192.168.1.5:63091 --
Aug 31 21:39:36 unused5 mDNSResponder[28]: 1 Questions
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 mail.munnik.net. Addr
Aug 31 21:39:36 unused5 mDNSResponder[28]: 1 Answers
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 TTL 86400 4 mail.munnik.net. Addr 192.168.1.121
Aug 31 21:39:36 unused5 mDNSResponder[28]: 1 Authorities
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 TTL 86400 15 munnik.net. NS ns.munnik.net.
Aug 31 21:39:36 unused5 mDNSResponder[28]: 1 Additionals
Aug 31 21:39:36 unused5 mDNSResponder[28]: 0 TTL 86400 4 ns.munnik.net. Addr 192.168.1.121
Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: DNSServiceQueryRecord(mail.munnik.net., Addr) ADD 4 mail.munnik.net. Addr 192.168.1.121
Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: Cancel 00000000 00000001
Aug 31 21:39:36 unused5 mDNSResponder[28]: 46: DNSServiceQueryRecord(mail.munnik.net., Addr) STOP
Aug 31 21:39:38 unused5 mDNSResponder[28]: 46: Removing FD

Results in the following in the cache:

Aug 31 21:39:42 unused5 mDNSResponder[28]: Slt Q TTL if U Type rdlen
Aug 31 21:39:42 unused5 mDNSResponder[28]: 72 107996 -U- Addr 4 mail.munnik.net. Addr 192.168.1.121



So in the fixed situation the TTL in the packet log nicely matches the MIN TTL, while in the error situation the TTL in the packet log does not match the TTL in the zone file. (in both cases the TTL in the cache does not match the TTL in the packet log).

Other than a variable overflow I do not see how to make sense of this ?

There is some library problem here. 10.4, 10.5, Ubuntu, Solaris and even Windows XP are able to handle have both my ISPs and my local DNS server in the DHCP set DNS configuration. 10.6 is blowing it. Using dig I get correct results but all of the other tools are not resolving correctly. I have tried ping, telnet, and ssh and none resolve localsys.localdomain.tld. Output from dig, anonymized by hand:

# dig localsys.localdomain.tld

; <<>> DiG 9.6.0-APPLE-P2 <<>> localsys.localdomain.tld
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43898
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;localsys.localdomain.tld. IN A

;; ANSWER SECTION:
localsys.localdomain.tld. 3600 IN A 192.168.202.11

;; AUTHORITY SECTION:
localdomain.tld. 3600 IN NS localsys.localdomain.tld.

;; ADDITIONAL SECTION:
localsys.localdomain.tld. 3600 IN A 192.168.202.11

;; Query time: 9 msec
;; SERVER: 192.168.202.11#53(192.168.202.11)
;; WHEN: Mon Aug 31 16:25:51 2009
;; MSG SIZE rcvd: 93

# scutil --dns
DNS configuration

resolver #1
domain : localdomain.tld
search domain[0] : localdomain.tld
nameserver[0] : 192.168.202.11
nameserver[1] : 207.172.3.8
nameserver[2] : 207.172.3.9
order : 200000

resolver #2
domain : local
options : mdns
timeout : 2
order : 300000

resolver #3
domain : 254.169.in-addr.arpa
options : mdns
timeout : 2
order : 300200

resolver #4
domain : 8.e.f.ip6.arpa
options : mdns
timeout : 2
order : 300400

resolver #5
domain : 9.e.f.ip6.arpa
options : mdns
timeout : 2
order : 300600

resolver #6
domain : a.e.f.ip6.arpa
options : mdns
timeout : 2
order : 300800

resolver #7
domain : b.e.f.ip6.arpa
options : mdns
timeout : 2
order : 301000