why do i get "connection refused" on some dns lookups?

Question

Level 1

48 points

why do i get "connection refused" on some dns lookups?

if i try to resolve certain hostnames (some google domains, in particular), i have problems.

on occasion, i believe this bug prevents me from being able to display google maps correctly. it also seems to make other pages load slowly or fail to load completely.

based on other comments i've seen elsewhere on the web, i believe this relates to cases when the resolver response is large (which tends to happen for google domains and akamai domains since they tend to have lots of alternative servers). when the response exceeds 512 bytes, i believe it falls back to using TCP. what's less clear from these other discussions is exactly why i'm having the problem and more importantly what to do about it. some other threads end with the recommendation to turn off IPv6. that is not an issue in my case. other threads devolve into theories about problems relating to different hardware devices (routers).

i'm using what i believe to be a pretty vanilla setup, running mac os x 10.6 server, with dns server turned on and using a forwarding address of that provided by my ISP. because this is happening over the lan (via an airport wireless basestation) it doesn't seem like there should be any issues of ports not being open or firewall issues or anything like that - but correct me if i'm wrong about that.

i see the following results when using (for example) the hostname www.google-analytics.com:

% dig @<my-osx-server> www.google-analytics.com

;; Truncated, retrying in TCP mode.

;; Connection to 192.168.2.103#53(192.168.2.103) for www.google-analytics.com failed: connection refused.

%dig @<my-isp's-dns-server> www.google-analytics.com

; <<>> DiG 9.6.0-APPLE-P2 <<>> +ignore @duff.homedns.org www.google-analytics.com

; (1 server found)

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31551

;; flags: qr tc rd ra; QUERY: 1, ANSWER: 17, AUTHORITY: 10, ADDITIONAL: 0

;; QUESTION SECTION:

;www.google-analytics.com. IN A

;; ANSWER SECTION:

www.google-analytics.com. 75504 IN CNAME www-google-analytics.l.google.com.

www-google-analytics.l.google.com. 105 IN A 74.125.226.163

www-google-analytics.l.google.com. 105 IN A 74.125.226.164

www-google-analytics.l.google.com. 105 IN A 74.125.226.165

www-google-analytics.l.google.com. 105 IN A 74.125.226.166

www-google-analytics.l.google.com. 105 IN A 74.125.226.167

www-google-analytics.l.google.com. 105 IN A 74.125.226.168

www-google-analytics.l.google.com. 105 IN A 74.125.226.169

www-google-analytics.l.google.com. 105 IN A 74.125.226.170

www-google-analytics.l.google.com. 105 IN A 74.125.226.171

www-google-analytics.l.google.com. 105 IN A 74.125.226.172

www-google-analytics.l.google.com. 105 IN A 74.125.226.173

www-google-analytics.l.google.com. 105 IN A 74.125.226.174

www-google-analytics.l.google.com. 105 IN A 74.125.226.175

www-google-analytics.l.google.com. 105 IN A 74.125.226.160

www-google-analytics.l.google.com. 105 IN A 74.125.226.161

www-google-analytics.l.google.com. 105 IN A 74.125.226.162

;; AUTHORITY SECTION:

. 72803 IN NS a.root-servers.net.

. 72803 IN NS b.root-servers.net.

. 72803 IN NS k.root-servers.net.

. 72803 IN NS e.root-servers.net.

. 72803 IN NS c.root-servers.net.

. 72803 IN NS l.root-servers.net.

. 72803 IN NS j.root-servers.net.

. 72803 IN NS h.root-servers.net.

. 72803 IN NS g.root-servers.net.

. 72803 IN NS i.root-servers.net.

;; Query time: 10 msec

;; SERVER: 192.168.2.103#53(192.168.2.103)

;; WHEN: Wed Apr 27 14:13:49 2011

;; MSG SIZE rcvd: 508

Mac OS X (10.6.7)

Posted on Apr 27, 2011 11:39 AM

Reply

Answer 1

MrHoffman

Level 10

146,299 points

Apr 27, 2011 12:51 PM in response to David Duff

First, remove the DNS forwarder setting within Mac OS X Server DNS set-up. You don't need that. You're running a real DNS server, so your server can chat directly with other servers. (Forwarders are useful for tasks such as DNS caching and DNS-based nanny filters, but otherwise just add another hop into the DNS translation overhead.)

Second, have a look at the DNS server logs for any related chatter in the system logs, such as for any advertised EDNS UDP packet size chatter. (If you see that, then your gateway or intervening router(s) or the target DNS server might have a configuration error or related limitation. And if you do see those EDNS errors, also see the OARC DNS Reply Size Test Server and related information.)

And for completeness, launch Terminal.app and toss the sudo changeip -checkhostname command at your server, just to see if your server thinks its own DNS is valid. You should get some chatter and an indication that no DNS changes are required. If not, then post the results and we'll have a look.

Reply

Answer 2

David Duff Author

Level 1

48 points

Apr 27, 2011 1:24 PM in response to MrHoffman

dns forwarder config makes sense for me based on tests that i did due to my being a fairly high-latency connection. although i could certainly turn it off if you think it bears on this specific issue.

didn't see anything informative in the logs. did "changeip -checkhostname" and all is well there.

here's some additional data:

sort of on a hunch, i clicked "allow zone transfers" in Server Admin.app for my main zone. queries that were previously failing started working.

however, to try to test/verify the theory, i turned zone transfer back off. the queries still worked. so this wasn't really conclusive.

i wasn't sure if there might be other external variables that were affecting my results (i.e. google possibly returning smaller responses), so i tried several queries that were previously failing and i saw the result (see below) which seems to indicate that tcp retry is now definitely working, whereas it wasn't before.

so my question: is it possible that my dns server somehow was not configured to listen over TCP initially and somehow the "zone transfer" option somehow fixed this - and that the fix survived even after the setting was reverted to its previous setting? or was this just a fluke?

$ dig mt0.googleapis.com

;; Truncated, retrying in TCP mode.

; <<>> DiG 9.6.0-APPLE-P2 <<>> mt0.googleapis.com

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63337

;; flags: qr rd ra; QUERY: 1, ANSWER: 17, AUTHORITY: 13, ADDITIONAL: 21

;; QUESTION SECTION:

;mt0.googleapis.com. IN A

;; ANSWER SECTION:

mt0.googleapis.com. 414151 IN CNAME mt.l.google.com.

mt.l.google.com. 16 IN A 74.125.226.171

mt.l.google.com. 16 IN A 74.125.226.172

mt.l.google.com. 16 IN A 74.125.226.173

mt.l.google.com. 16 IN A 74.125.226.174

mt.l.google.com. 16 IN A 74.125.226.175

mt.l.google.com. 16 IN A 74.125.226.160

mt.l.google.com. 16 IN A 74.125.226.161

mt.l.google.com. 16 IN A 74.125.226.162

mt.l.google.com. 16 IN A 74.125.226.163

mt.l.google.com. 16 IN A 74.125.226.164

mt.l.google.com. 16 IN A 74.125.226.165

mt.l.google.com. 16 IN A 74.125.226.166

mt.l.google.com. 16 IN A 74.125.226.167

mt.l.google.com. 16 IN A 74.125.226.168

mt.l.google.com. 16 IN A 74.125.226.169

mt.l.google.com. 16 IN A 74.125.226.170

;; AUTHORITY SECTION:

. 68603 IN NS i.root-servers.net.

. 68603 IN NS d.root-servers.net.

. 68603 IN NS m.root-servers.net.

. 68603 IN NS a.root-servers.net.

. 68603 IN NS h.root-servers.net.

. 68603 IN NS k.root-servers.net.

. 68603 IN NS c.root-servers.net.

. 68603 IN NS j.root-servers.net.

. 68603 IN NS g.root-servers.net.

. 68603 IN NS l.root-servers.net.

. 68603 IN NS f.root-servers.net.

. 68603 IN NS e.root-servers.net.

. 68603 IN NS b.root-servers.net.

;; ADDITIONAL SECTION:

a.root-servers.net. 376426 IN A 198.41.0.4

a.root-servers.net. 430718 IN AAAA 2001:503:ba3e::2:30

b.root-servers.net. 379968 IN A 192.228.79.201

c.root-servers.net. 379925 IN A 192.33.4.12

d.root-servers.net. 387256 IN A 128.8.10.90

e.root-servers.net. 386496 IN A 192.203.230.10

f.root-servers.net. 541901 IN A 192.5.5.241

f.root-servers.net. 557479 IN AAAA 2001:500:2f::f

g.root-servers.net. 388515 IN A 192.112.36.4

h.root-servers.net. 550658 IN A 128.63.2.53

h.root-servers.net. 572036 IN AAAA 2001:500:1::803f:235

i.root-servers.net. 540553 IN A 192.36.148.17

i.root-servers.net. 579288 IN AAAA 2001:7fe::53

j.root-servers.net. 547773 IN A 192.58.128.30

j.root-servers.net. 562734 IN AAAA 2001:503:c27::2:30

k.root-servers.net. 543114 IN A 193.0.14.129

k.root-servers.net. 556107 IN AAAA 2001:7fd::1

l.root-servers.net. 365274 IN A 199.7.83.42

l.root-servers.net. 368148 IN AAAA 2001:500:3::42

m.root-servers.net. 381970 IN A 202.12.27.33

m.root-servers.net. 400045 IN AAAA 2001:dc3::35

;; Query time: 8 msec

;; SERVER: 192.168.2.103#53(192.168.2.103)

;; WHEN: Wed Apr 27 16:09:47 2011

;; MSG SIZE rcvd: 961

Reply

Answer 3

David Duff Author

Level 1

48 points

Apr 27, 2011 1:26 PM in response to MrHoffman

also, here's the oarc reply test results:

$ dig +short rs.dns-oarc.net txt

rst.x3827.rs.dns-oarc.net.

rst.x3837.x3827.rs.dns-oarc.net.

rst.x3843.x3837.x3827.rs.dns-oarc.net.

"68.87.71.233 DNS reply size limit is at least 3843"

"Tested at 2011-04-27 20:00:24 UTC"

"68.87.71.233 sent EDNS buffer size 4000"

Reply

Answer 4

MrHoffman

Level 10

146,299 points

Apr 27, 2011 4:36 PM in response to David Duff

What might be in the DNS logs around the failures? (I couldn't conclusively rule out a flaky WiFi with what's posted here.)

Do you have any DNS servers referenced (other than 192.168.2.103) anywhere that DNS servers can be specified? Mixing DNS disparate servers including referencing ISP servers from clients can cause issues.

(And FWIW, for local DNS, I'd likely only have zone transfers lit if there are secondaries around, and if the server was behind a firewall or otherwise locked down.)

If you want to test local queries and local caching, you will want to use:

dscacheutil -q host -a name host.example.com dscacheutil -q host -a ip_address 172.16.1.2

And one of the commands that can be used flush the local DNS cache is this:

sudo killall -HUP mDNSResponder

Also look at shutting off IPv6, and run a test here.

And FWIW, 192.168.0.0/16 is a bad block for using VPNs, though you're not in the worst of the subnets in that block.

Reply

Answer 5

David Duff Author

Level 1

48 points

Apr 29, 2011 7:28 AM in response to MrHoffman

DNS logs? no - i see nothing informative in the DNS logs on my server.

Any DNS servers referenced other than my local server? No. Unless a browser or other app is not updating when i move a client (laptop) around on the network.

I don't need zone transfers for my setup. what i found slightly interesting is that behavior seemed to change when i turned it on. i've turned it back off; the behavior stayed the same. so i'm thinking this might have been a red herring.

i have no particular interest in testing local queries and local caching, or flushing the local cache since evidence points to the problem being with my client machine and it's ability to communicate with my server.

i'm sure IPv6 is not a factor. it is shut off on both client and server.

small mystery: netstat doesn't seem to indicate that my server is listening on tcp port 53. i do see that something is listening on udp port 53. even when i turn on zone transfers, i still don't see anything listening on tcp 53. could it be that the server waits for a request over udp before it starts listening on tcp?

Reply

Answer 6

May 22, 2011 8:57 AM in response to David Duff

I suspect I know what is happening here, but I don't know how to fix it.

The giveaway is the

;; Truncated, retrying in TCP mode.

It means that the response to the original UDP based DNS query was bigger than UDP will cope with. The result should be the query being repeated over TCP.

The next message

;; Connection to 192.168.2.103#53(192.168.2.103) for www.google-analytics.com failed: connection refused.

means that the mac server refused the TCP connection to it's DNS server.

I'm looking into how to sort that out.

FWIW in my case, I have a home machine that goes to my ISP's DNS directly (www.youtube.com resolves fine for example), and work machines that go via a snow leopard server's DNS (to resolve the names of work servers on my network) (www.youtube.com borked on requests from NSLOOKUP remotely - connection refused) but NSLOOKUP works locally on the server)

Reply

Answer 7

MrHoffman

Level 10

146,299 points

May 22, 2011 10:20 AM in response to blindjustis

dig and whois and other such effectively test the DNS server, but will bypass the local Mac OS X DNS pieces. For those queries, the dscacheutil is your tool of choice.

Get rid of the forwarders. That is entirely unnecessary. Your DNS server can query other DNS servers directly.

Also look for the usual sorts of network routing errors, including address collisions.

And no EDNS errors or related in the log, from the OARC write-up?

Reply