OSX Server Caching not working after updates

Question

Level 1

4 points

OSX Server Caching not working after updates

It's been low on my priority list, but the OSX server we run in house for caching seems to have stopped caching; from what I recall, it was related to updating the software to 10.11.4 from whatever it was before, and the Server software from whatever was installed to 5.1. Updates to both to the latest versions available right now haven't helped. It's kind of frustrating, but all it caches now are requests it makes... for itself. No client requests are seen. It did previously work.

There don't seem to be any useful diagnostics I can find to work out why the heck this suddenly doesn't work.

Platform: Mac Mini, "late 2014", bought this year. OSX currently on 10.11.5; Server on 5.1.5

Network setup: Multiple public IP addresses; multiple subnets.

Clients do not use same IP as server.

Full site IP allocation listed in _aaplcache._tcp DNS TXT records (plural - because I've done it for the DNS records for each client LAN DNS subdomain) using the prs type; DNS is on Windows Server 2012R2. No leading length characters, as they were not specified when the OSX Server Caching setup generated the requisite DNS records.

No firewall sits between the server and the clients on the LAN (only a L3 switch), but obviously, if client traffic leaves our network it passes through a firewall. No changes to the ruleset since it last worked.

The Mac Mini has an interface (with IP address and matching DNS record) in each client subnet (using VLANs), but it appears to register with Apple's servers using its main wired LAN IP (which clients can reach) with Apple's servers.

Mac Mini connects through gigabit ethernet.

Wireless connectivity is through a mixture of HP MSM and Ubiquiti access points; neither system seems to result in client traffic.

My understanding of the protocols involved are something along these lines.

-OSX server registers with Apple, using its normal connection; passes local LAN IP for the cache clients to use; may pass DNS TXT records to assist in "bootstrapping" clients, or may use the settings input in the cache server setup.

-Clients use the DNS TXT record to inform Apple's global servers that they need the cache server's IP address.

-Apple returns the local LAN IP address.

-Client speaks to caching server.

-Profit.

I can certainly see the first part happening - the Debug.log shows apparently successful registration; the cache service in the server applet is green - but no client requests seem to happen.

Blanked out IP addresses are correct.

Things are green and seem right.

Looks like it should be happy.

IPs are correct, and match DNS records.

Help! What other steps can I take to diagnose - and fix - this issue? The Apple help documentation isn't really very helpful (I understand what it says), but it doesn't really give sysadmin level insight into what to do when it doesn't "just work".

Thanks in advance for any insights into how to properly debug and fix this.

Mac mini (Late 2014), OS X El Capitan (10.11.5), OSX Server 5.x caching

Posted on May 24, 2016 4:06 AM

Reply

Answer 1

KWCIT Author

Level 1

4 points

May 24, 2016 4:14 AM in response to KWCIT

...clients can and do fetch software off the internet (which we want to avoid as much as possible).

Cached data amount is so low as I purged it (reset button) in case that was randomly an issue. No joy. Was previously over 300Gb in size, across many content types.

If it's not clear, both clients and servers use NAT, but different IPs are used, depending on the subnet they come from. This has previously worked.

Reply

Answer 2

May 24, 2016 7:37 AM in response to KWCIT

I wonder if the problem is the use of the non-default "all networks" setting. Please try:

1. Server > Caching > Edit Permissions... > Cache content for clients connecting from > only local subnets.

2. Reboot a client (on the same subnet as the server), which forces it to look up local servers again. Do not reboot or restart the server.

3. Try downloading something from that client.

If "only local subnets" is too limiting for you, I wonder if this would work:

1. Server > Caching > Edit Permissions... > Cache content for clients connecting from > only some networks, define some networks, and make sure the "Only cache content for clients on these networks" box is checked.

2. Reboot a client (on one of the defined networks), which forces it to look up local servers again. Do not reboot or restart the server.

3. Try downloading something from that client.

Please reply with your results.

Reply

Answer 3

KWCIT Author

Level 1

4 points

May 25, 2016 11:57 PM in response to More Broccoli Please

Thanks for your input More Broccoli Please.

The non-default settings are supported by Apple, *and* required on my network. (the OSX server and the clients all use different public IPs).

I tried your second suggestion, including the various subnets we require to be served by this caching server.

No requests from clients yet, and there would have been many reboots since I made this change 2 days ago on your suggestion.

Any other ideas, particularly how I can really dig "under the hood" to figure out why the cache requests from clients just don't seem to happen?

Reply

Answer 4

Leopardus

Level 4

1,303 points

May 26, 2016 2:08 AM in response to KWCIT

To default value to see the individual IP addresses and port number of a client requesting a package is set to false. You could enable logging this by using

sudo serveradmin settings caching:LogClientIdenty = true

and authenticate. To reverse that, one simply set the value again to false.

To have an overview of how your caching is presently used you can use

sudo serveradmin fullstatus caching

I really would recommend the iBooks on El Capitan by Reid Bondonis, there are a lot of handy little tips that one can use in Server.

HTH

Leo

Reply

Answer 5

May 26, 2016 3:34 AM in response to KWCIT

I have been fighting this issue for months and there IS NO resolution!!!!!

The MacApp Store/Caching Service connection is just plain broken!!!

Changing network settings, resetting server, changing default service ports, etc.,

etc., etc. did not work!!!!!

It is a service that either works like a champ or does not work at all, with no solutions.

Reply

Answer 6

KWCIT Author

Level 1

4 points

May 27, 2016 1:30 AM in response to KWCIT

I worked it out. Somehow, a "typo" crept into the DNS records that are required to make this work. Not too sure how, or quite why I didn't notice it until *after* a complete nuke and pave of the mac mini...

Essentially, we were missing an entire octet of the public ip address (i.e. the record was nonsense) range on the prs record (which obviously breaks things quite badly for obvious reasons).

I am now seeing a fair amount of caching traffic happening, which is good.

Check those DNS records, and check what DNS records your clients are getting for those _tcp._aaplcache TXT records...

Thanks for the ideas.

Reply

Answer 7

Leopardus

Level 4

1,303 points

May 27, 2016 3:23 AM in response to KWCIT

Glad that it is working!

It has been pointed out countless times before that working DNS is one of the fundamentals which makes Server tick.

Have fun

Leo

Reply

Answer 8

May 27, 2016 2:34 AM in response to KWCIT

KWCIT wrote:

I worked it out. Somehow, a "typo" crept into the DNS records that are required to make this work. Not too sure how, or quite why I didn't notice it until *after* a complete nuke and pave of the mac mini...

Essentially, we were missing an entire octet of the public ip address (i.e. the record was nonsense) range on the prs record (which obviously breaks things quite badly for obvious reasons).

I am now seeing a fair amount of caching traffic happening, which is good.

Check those DNS records, and check what DNS records your clients are getting for those _tcp._aaplcache TXT records...

Thanks for the ideas.

Where does one find this information in an attempt to see if this is the issue I am having?

Most importantly, how does one fix it?

Although I doubt that it will help since iCloud Caching and Caching with iTunes and iOS device

app downloads work. It is the MacApp Store/Caching Service connection that is broken for me,

but hey, I will try anything at this point.

Reply

Answer 9

KWCIT Author

Level 1

4 points

May 27, 2016 3:45 AM in response to woodmeister50

Use DNS lookup tools on a client to query the _aaplcache._tcp TXT record.

I'm very Windows-y, I'm afraid; nslookup record syntax is:

nslookup -q=TXT _aaplcache._tcp.<your domain>

If it exists, you will have the PRS record returned - make sure it matches the first and last public IP you expect in a contiguous range. If you have multiple ranges, you will need multiple PRS records - read about the chaining syntax in this record type - it is documented.

In Ubuntu, you'd use dig, which should work in OSX as well:

dig -t txt _aaplcache._tcp.<yourdomain>

Similarly, you should see the prs record returned. If you have multiple subdomains with _aaplcache records, query each one in turn.

Are you using the caching server or the other mac update server (hides under Advanced in the server app left hand menu bar)? I seem to recall reading you can use one or the other, not both? I also distinctly remember reading that the caching server does NOT cache iTunes downloads.

If your network is a "normal" SOHO network, with a single border router with a single public IP address everything NATs out on, the caching server should "just work" (without the DNS setup), so long as your client IP address range(s) is(are) set up in the caching server.

Reply

Answer 10

KWCIT Author

Level 1

4 points

May 27, 2016 4:11 AM in response to KWCIT

... and fixing it is a matter of getting your DNS server (which is normally your "router" in a SOHO network) to return the correct results. Most SOHO routers don't know what a TXT record is - but you also typically don't need a TXT record with a SOHO router, because single IP NAT should "just work" (because the cache server's public IP === your clients public IP).

The caching service doesn't cache *everything*; check the specs/documentation to see what it ought to cache. Content types supported by the Caching service in OS X Server - Apple Support

Reply