Apple Event: May 7th at 7 am PT

Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

woodmeister50 Author

Level 6

19,723 points

Caching Server service still broke for Mac App Store apps

Well, I thought I had found a fix to my issue of downloads from the Mac App Store not caching.

WRONG!

I explicitly set the flags, via terminal commands, for AllowPersonalCaching and AllowSharedCaching

as the two flags were not present in config.plist for the service. I then downloaded an app from the

Mac App Store on one of my Macs and it did actually get cached and served from the Caching Service.

Well, thought all was good, until today!

Saw that I had an update for 1Password app. So downloaded it from the MacApp Store onto my iMac.

Then I thought, better check my MiniServer to see if the Caching Service was still working for MacApp

Store downloads. Guess what? NOT!!! Did the update on two other Macs and the Mini Server and none

of them made the Caching Service budge!

However, all the Macs do see the caching server and a look in the log shows that the Caching Service is working

for local iCloud caching as when each Mac was powered up for the update I was attempting, there were multiple

PUTs and GETs for cloudd/xxx.xx.

Now, on top of that, the formerly working iTunes Store app update caching when downloading on the Macs is also

not working any more!!! It did however, serve an app directly to an iDevice that I tested.

So, has the Caching Service just basically a useless consumption of memory now in terms of working with Mac?

Has it turned into an iDevice only service? I have tried re-starting the service, shutting down the Server app, rebooting

the server all to no avail.

All the functionality of the service was fine under Yosemite. Somewhere along the line though, either El Capitan updates

or Server App updates have some how borked this service.

So, any clues or do people simply not use this service anymore and no one cares?

OS X El Capitan (10.11.4), Home Server

Posted on Apr 5, 2016 8:43 AM

45 replies

May 6, 2016 6:40 AM in response to woodmeister50

I confirm word by word the problem that woodmeister50 is reporting: osx application updates don't go via Caching Server if you have 10.11.4 installed...

Please note that if you update from 10.10 you are actually downloading and installing the full application. Recently a new "incremental update" feature has been implemented and it may have broken caching...

On my 10.11.4, App Store application reports a 22Mb application update but the download counter stops before reaching that number...

So, if you have devices with 10.11.4, please ask the lan people to sniff the lan traffic to 17.253.0.0/16 network and you will see that some devices are downloading updates directly from Apple. You won't and can't see these downloads in other ways (except if you do them from your workstations and have a look at Debug.log).

It seems that iOs updates are ok, but I couldn't fully test them yet.

May 6, 2016 7:24 AM in response to mercurialuser

Just tested. Out of 6 iOs application updates, only 3 went via Caching Server. Notably, Acrobat Reader and Chrome went directly to Apple servers. Tripadvisor used the cache.

woodmeister50 Author

Level 6

19,723 points

May 7, 2016 4:01 AM in response to Leopardus

I found the problem with the empty database file. Unlike your configuration,

my Caching Service is using an external drive and all that data is located there.

I haven't had time to re-investigate but I am lead to believe that Mercurialuser may have

some substance though, the issue has been around before 10.11.4. When

MacApp Store apps are updated, there is no log activity what so ever on the server

and also monitoring the server during the act of updating an app on another

Mac shows no additional disk or network activity than what one would see when

just idle.

As I said though, I will look at the database in detail but I suspect based on other

data that I will still find MacApp Store apps updates not present.

So, the current state is:

iCloud caching -> working from all devices (at least the log is showing activity).

iDevice direct app updates -> downloads cached or served from cache

iDevice app updates via iTunes on a Mac -> downloads cached or served from cache

MacApp Store OS X component updates -> cached and served from cache for a single update (Camera Raw in this case).

Won't know if it is still the case until another update to an OS X component

comes along. NOTE: This worked after a change to port the config file.

MacApp Store app updates -> Not cached or served from cache or even recognized as being requested. The most recent

of which was Xcode which was sizable enough to monitor.

woodmeister50 Author

Level 6

19,723 points

May 11, 2016 3:49 AM in response to woodmeister50

Well, I guess there is no solution and it is just broken software, either at the server end

or client end or both!!!

woodmeister50 Author

Level 6

19,723 points

May 18, 2016 3:36 AM in response to woodmeister50

Well, the recent updates to Server and OS X have not changed a single thing.

Even did a cache reset after the update.

It is looking more and more that it is a client side issue since no log anywhere

on the server shows any hint that a Mac has made an update request from the

MacApp Store.

woodmeister50 Author

Level 6

19,723 points

Jun 2, 2016 11:27 AM in response to woodmeister50

I'm back again.

Had some time and thought I would try one last ditch effort to see what is going on with

MacApp Store updates not working. On two Macs I had 3 updates available. One one Mac,

updated an app and checked the Caching Server log in the GUI as well as debug.log. Nothing

served, nor was anything even requested, i.e. no mention of even an attempt at a request.

Next, I figured I would run Wireshark while doing an update to an app and look for traffic filtered

for my server IP.

I then attempted an update with the next app in line. Well, guess what, the app update request

was received and served via web and cached in the server. There was all manner of traffic in Wireshark

when this took place. Did something fix itself? NOT!!!!!

When requesting the 3rd app update, no request seen by the Caching Service and looking at Wireshark,

the was nary a peep from the Mac to the server.

On the other Mac, this time started with Wireshark running. Of the three apps that were updated, only one

was serviced by the Caching Server, but not the same app that was served on the first Mac but a different one.

Again, when the apps were not served, there was not network traffic from the Mac to the Server IP, but multitudes

of data when the app was served.

So, a request to update an app on one client worked with the Caching Server, the same app didn't on

another for the same app. And, only 1 out of the three updates were serviced on either machine.

It is almost looking as if the MacApp Store app just randomly decides on its own when it wants to make

a request from the server or just simply ignore the fact that the Caching Server is even there!!!!!

And to re-iterate, iCloud Caching, iDevice app request Caching, and iTunes Store on Mac app caching is

working. Why is the MacApp Store hosed????

Jun 2, 2016 6:30 PM in response to woodmeister50

Hi there, I've been tracking your updates, that is really strange the Mac app updates aren't working.

Shot in the dark question, is your Caching server running DHCP/DNS for your network? (I can't remember)

If it's not running those, try and see if you can encourage your Mac clients to send all domain lookups through the DNS on the caching server

Leopardus

Level 4

1,303 points

Jun 3, 2016 12:18 AM in response to woodmeister50

Have you checked in the db what has happened recently wrt to the OS X updates, if you had any? Think there were also some iOS 9 stuff to be updated as well?

There is no client configuration to be done at all. Everything happens with server and its unique GUID code which it receives and is registered with Apple's Servers, which 'appears' as a ZGUID code for the specific software, referencing a unique location on your HD. You would thus receive a verbatim copy of the software, exactly as if it comes from Apple.

What is essential, however is that DNS must work correctly for your network. A tiny typo can have a massive negative influence in DNS.

Leo

Jun 3, 2016 1:34 AM in response to woodmeister50

I did a test today, after days in which I have good caching results.

I have 2 Caching servers, mini1 and mini2, in peer. On both I install a lot of "free" apps, almost everything in the "Great Free Apps & Games" panel. So, to be clear, my "user" and my "server" are on the same phisical machine. I don't own another mac to do other tests, at the moment.

Today I had 2 updates on both servers: iTunes 12.4.1 and Simplenote.

I started with iTunes. I was very surprised that both retrieved the software from cache ! From their own cache. Each one had a copy of the software.

I then updated Simplenote on mini1 and I was a lot more surprised ! mini1 asked mini2 if it had the update, and mini2 answered it didn't. mini1 downloaded from apple. When I updated mini2, it asked mini1 if it had the update and of course it had it ! So the software was transferred from mini1 to mini2 and the installed. It's the first time in my tests that a software update on mac osx fully uses the Caching servers - my first tests were in January 2016.

About iTunes update. I wrote that mini1 and mini2 both had the software in their cache. I browsed the logs and saw that this morning one client asked for the update to mini1 (that downloaded from Apple) then another client asked to mini2 which received the update from mini1 and stored it in the local cache. It seems to me that this in a undocumented caching policy: data is be duplicated on peering servers. It's not necessarily bad, it's a policy with pros (imagine a broken server, you may have duplicated ready-to-serve contents) and cons (you have less available storage space).

I'm sure that when I will update other apps I won't get them all from the cache. It's Apple that tells the clients from where to update, and I think it also depends on how the update is packaged, the version of osx you have installed and a lot of other variables.

Jun 3, 2016 2:25 AM in response to woodmeister50

@woodmeister50

I've been having good caching results in the last few days. I don't know exactly why but the stats are a lot better.

I asked the wifi people to extend the lease time of dhcp to minimum 7 days... my dream is to assign every user a "static" ip address... in this way they present themselves to the server with the same IP. I don't know if this helped or if it is just a casualty, but it's what we did.

Jun 3, 2016 3:21 AM in response to mercurialuser

Just to continue on "something is strange about mac osx updates" thread, one mac propose an update to Twitter 4.1.2, published on Jun 2, the other mac has 4.1.1 and no update available.

I update Twitter to 4.1.2 and Slack to 2.0.3 and both DID NOT ask the Caching Server.

woodmeister50 Author

Level 6

19,723 points

Jun 3, 2016 3:29 AM in response to chattphotos

My DHCP/DNS setup has not changed for the last several years and is managed

by my Airport Extreme Base Station (basically since I first purchased it).

If you follow through the thread, you will see that all worked just fine until somwhere

in OSX 10.11.x and Server 5.x update chain and was flawless with 10.10.x and Server

5.x or 4.x.

And, it is only the MacApp Store that is a problem!!!

iOS app updates, whether via iDevice or Mac/iTunes work flawlessly. iCloud caching is

responding for all clients. All of which are verified in the logs yesterday by doing iOS updates

via iDevice or iTunes on Mac. iCloud caching is responding as I made changes in iCloud Drive

or apps that utilize it and see the events in the log.

As of yesterday, as stated, the MacApp Store app caching is intermittent at best and when not

cached, the client issued no request to the server, verified by no packets sent from client to server at all.

A bit more detail in the scenarios yesterday, in one case it was the "middle" update that worked and in the other

scenario it was the first update that worked.

Leopardus

Level 4

1,303 points

Jun 3, 2016 10:55 PM in response to mercurialuser

mercurialuser wrote:

@woodmeister50

I've been having good caching results in the last few days. I don't know exactly why but the stats are a lot better.

... my dream is to assign every user a "static" ip address... in this way they present themselves to the server with the same IP. I don't know if this helped or if it is just a casualty, but it's what we did.

You should be able to 'reserve' an IP for the MAC address and/or the 'Bonjour' name, which would in essence give you a 'static' address per device.

HTH

Leo

woodmeister50 Author

Level 6

19,723 points

Jun 5, 2016 3:43 AM in response to mercurialuser

mercurialuser wrote:

Just to continue on "something is strange about mac osx updates" thread, one mac propose an update to Twitter 4.1.2, published on Jun 2, the other mac has 4.1.1 and no update available.

I update Twitter to 4.1.2 and Slack to 2.0.3 and both DID NOT ask the Caching Server.

Welcome to the club! I have come to the conclusion that somewhere in the upgrade paths

from OS X 10.10 or 10.11.0 and Server 5.0 to current there have been changes that have

created random breaking of the MacApp Store Caching service. This service worked

perfectly previous to the path I have mentioned. The randomness of the working and not working,

especially with my recent tests says the whole concept has become borked!!!!

woodmeister50 Author

Level 6

19,723 points

Jun 8, 2016 9:29 AM in response to woodmeister50

I give up!!!!

In terms of the MacApp Store, the Caching Service is just totally hosed and decides to

work when it feels like (or not). 4 Macs, 4 MacApp Store updates on each (one of which

was iTunes 12.4.1), only one was even recognized as requesting the updates and the apps

requested by it were cached. On the other three, including the server that had the Caching Service,

it was like they did not even exist!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! By the way, the one that worked,

was one that was totally ignored in the last post. The service seems to just randomly chose

who it will service and how much it will service.

On two Macs, iOS updates were requested via iTunes and both machines were recognized

and apps were served or cached, one of which was the server that the Caching Service is running on.

Bottom line, the MacApp Store part of the service is just TOTALLY unreliable, flakey, works when it wants to,

and choses randomly to do so!!!!!!!!

Caching Server service still broke for Mac App Store apps