mDNSresponder cache policy for REFUSED answers

Question

serge-r Author

Level 1

5 points

mDNSresponder cache policy for REFUSED answers

Hello!

A few weeks ago, I began researching a problem related to Outline VPN and Cloudflare DNS over TLS on Apple devices.

You can find details here https://community.cloudflare.com/t/1-1-1-1-1-0-0-1-dns-over-tls-limitations/643561

and here https://github.com/Jigsaw-Code/outline-apps/issues/1980

This issue affects iOS and MacOS devices and is related to the DNS mechanism. Sometimes, Cloudflare servers respond with REFUSED code to DNS queries. It seems like mDNSResponder is caching this answer because I don’t see any packets in the traffic dump when I repeat the queries.

Here is a sample of mDNSResponder logs when I received a REFUSED answer:

2024-04-18 19:42:43.901120+0700 0x24b132   Default     0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q64229] Received acceptable 468-byte response from BBhjVjYR over TLS via utun7/32 -- id: 0xAB9A (43930), flags: 0x8185 (R/Query, RD, RA, Refused), counts: 1/0/0/1, BBgKPenV IN A?, . OPT 1232 0 {Padding, [422 B]}
2024-04-18 19:42:43.901319+0700 0x24b132   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] Penalizing server BBhjVjYR for 60 seconds
2024-04-18 19:42:43.901394+0700 0x24b132   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q64229] Querier concluded -- reason: response
2024-04-18 19:42:43.901532+0700 0x24b132   Default     0x0                  389    0    mDNSResponder: [com.apple.mDNSResponder:Default] [Q64229] Handling concluded querier: BBgKPenV A IN
2024-04-18 19:42:43.901792+0700 0x24b132   Default     0x0                  389    0    mDNSResponder: [com.apple.mDNSResponder:Default] [R101936->Q64229] DNSServiceQueryRecord(<mask.hash: 'CqqpDNZbd5XmCw8ZIscR7Q=='>(11e4ef85), A) RESULT ADD interface 0: (mortal, DNSSEC Indeterminate)<mask.hash: '3QVv2uzZDFYViunWAFfkYA=='>
2024-04-18 19:42:43.933506+0700 0x24b132   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] Unpenalizing responsive server BBhjVjYR
2024-04-18 19:42:43.933578+0700 0x24b132   Info        0x0                  389    0    mDNSResponder: [com.apple.mdns:resolver] [Q41448] Querier concluded -- reason: response

Here's an attempt to resolve the domain again:

➜  ~ dns-sd -Q  riotgames.com                                                                                                                                       
DATE: ---Thu 18 Apr 2024---
19:43:06.431  ...STARTING...
Timestamp     A/R  Flags         IF  Name                          Type   Class  Rdata
19:43:06.433  Add  40000002       0  riotgames.com.                Addr   IN     0.0.0.0    No Such Record

As a result, all apps using the system resolver are unable to connect to this domain for some time.

Questions:

a) Is there a way to decrypt mDNSResponder logs? Currently, I can’t see the queries and responses, only some hashes.

b) What does "Penalizing server ... for 60 seconds" mean? Why doesn’t mDNSResponder try to use other resolvers during this period? Is it related to the DoT mechanism?

c) I’m not sure if caching REFUSED answers is normal behavior; I couldn’t find any RFC about it. Is there any documentation about how mDNSResponder works, or a way to avoid caching these answers?

d) I tried to set up a MitM (Man-in-the-Middle) proxy to catch the DNS responses and found that MacOS is unable to switch to DoT for my proxy, even though I have all certificates in the trusted chain and the kdig utility works normally. So, does mDNSResponder perform SSL pinning or something similar to ensure that certificates match the resolver addresses for DoT?

Thank you for your responses. If you need any detailed information, I can provide dumps and logs.

MacBook Pro 16″, macOS 13.3

Posted on Apr 18, 2024 6:10 AM

Reply

Answer 1

Apr 18, 2024 7:47 AM in response to serge-r

You have lots of different pieces active here.

Private Relay already uses ODoH for HTTP, and its server can be configured.

As for VPNs, either the two-hop Tor for HTTP provided by Private Relay, or run your own VPN and your own Algo server. VPNs can all have their own issues.

ODoH deliberately fragments the DNS traffic to make recovery more difficult. See below for packet capture examples.

You’re also seemingly also operating these ODoH changes in an online gaming environment, an environment which can be particularly skeptical about network changes and anything that looks like shenanigans. For assistance with not getting mis-detected by any anti-cheat mechanisms, contact Riot support and discuss compatibility of your ODoH and VPN and CloudFlare DNS efforts with them, too.

And if Cloudflare OdOH is refusing your DNS queries, that is their prerogative, and any block is seemingly best discussed with them, but I’d wonder this is error is originating from Riot DNS servers. (Not having dig’d their DNS server configuration.)

PS: Cloudflare client and server source code at Github, related info, and packet capture info: https://www.keysight.com/blogs/en/tech/nwvs/2021/03/11/oblivious-doh-enhanced-dns-privacy

Reply

Answer 2

serge-r Author

Level 1

5 points

Apr 18, 2024 11:25 PM in response to MrHoffman

Hello MrHoffman and thank you for your response.

It seems I should provide more details because there appears to be some misunderstanding.

The issue isn't with a particular domain, such as riotgames.com, which is just an example. The primary concern is with the users of our mobile application. We've received numerous tickets reporting problems loading the mobile app when Outline VPN is activated.

I conducted some investigations and found that Outline VPN triggers DNS over TLS (DoT) to Cloudflare because Cloudflare's address is hardcoded in the Outline client and listed first. From my tests, I can confirm that the problem does not originate from the VPN tunnel or network quality. Typically, VPN servers assign clients a single IP (due to NAT), and I discovered that as the number of DNS queries via DoT (and perhaps DNS over HTTPS, DoH) to Cloudflare increases, I could receive a REFUSED response for any domain. This suggests that Cloudflare might have per-IP limitations for DoT queries.

I have inquired about these limits with Cloudflare but have yet to receive a response. I also asked the developers of Outline about the possibility of randomizing DNS servers in the client for each connection, and I am currently waiting for their response.

One last point of uncertainty: I'm not sure if caching a REFUSED response and ignoring other configured resolvers by mDNSResponder is normal behavior or a bug. I hope to hear what Apple developers think about this issue.

Reply

Answer 3

Apr 19, 2024 8:59 AM in response to serge-r

I had thought this project had lots of active pieces, and now that I’ve done a little research on an unfamiliar VPN, realize it’s still more complex, and with other and far larger entanglements awaiting.

I’d strongly encourage a discussion with your organization’s legal advisors here particularly around the “widely used” aspects this VPN and its associated proxy servers, and would seriously also consider reworking the app to detect this VPN service and reporting a diagnostic about potential issues if feasible, of capturing this error and reporting it with some information that can explain what probably happened to the user (an unsupported network configuration), or would explicitly document the incompatibility if not.

Reply

mDNSresponder cache policy for REFUSED answers

Similar questions