App Store apps do not install properly on a fresh install of High Sierra on internal SSD or HDD

The database shared from softmusic back on November 26 is the one that worked for me. The computer I was working on needed iWork apps. Not only did it allow them to launch, it allowed the App Store to download the proper version. https://discussions.apple.com/thread/252037712?answerId=253981944022#253981944022

http://www.mediafire.com/file/m2ky3mrnon6jy49/valid.sqlite3.zip/file

johnno_uk wrote:

I could be wrong on all this off course but sending an empty update looks suspiciously like the first step in a plan towards fixing at least one category of user, those who install a fresh copy of High Sierra today or later on. Like I say though I think they'll bundle those that did an install before v146 into the same bucket and tell them just to reinstall because it's easier for them to do it that way.

I think I actually understand you here. In other words, they fixed something on the back end, and simply telling people to reinstall their system — because it's easier for them and I actually agree, remember I work with a lot of technically unsophisticated users —will cause whatever that back-end change was to take effect. ("Reinstall the OS" is a knee-jerk response Apple techs have to almost any problem, a long-standing gripe of mine.)

It's probably academic at this point, but if you look at https://discussions.apple.com/thread/252037712?answerId=254019170022#254019170022 where I posted a Finder screenshot of my backup database and the current database, there's a slight size difference in addition to the updated version number. If anybody wants copies of both to compare them in case the information would be helpful, let me know.

FrancoisQC wrote:

So where do we get this donor DB (I like your term btw!) ?

The database from /Library/Keychains/crls/? (not the one from inside the bundle). Well, softmusic posted the one I used on my client's computer about a week ago https://discussions.apple.com/thread/252037712?answerId=253981944022#253981944022

The link to the file is http://www.mediafire.com/file/m2ky3mrnon6jy49/valid.sqlite3.zip/file

This is the one from my computer which is from a pre-existing High Sierra install and has been working fine. https://www.dropbox.com/s/56nzoo5xpo9tcrw/valid.sqlite3.zip?dl=0

There will be more than meets the eye to all this. I now suspect that on 12th Nov there will be a pair of updates which revoke Apple's certificates by accident and quickly un revoke them.

Whilst the problems in this article mention an OSCP server meltdown ...

https://tidbits.com/2020/11/13/apple-network-failure-destroys-an-afternoon-of-worldwide-mac-productivity/

... Nobody appears to be asking why all those Macs were making OCSP requests in the first place. Really Apple just said, 'Yeah we do that', people said 'What about privacy', Apple said 'We'll use encryption more' and everyone said 'OK then'.

The article does have this in it though ...

"What prevented ocsp.apple.com from responding? I doubt Apple will ever share detail"

I think at least for an app launch these OSCP requests are only supposed to happen as a fallback incase the local database has an invalid revocation in it. High Sierra doesn't do this fallback (but using an app like WhatsYourSign effectively makes that happen) but I think later versions of the OS do this automatically. I don't think it's normal for every app launch (or even a launch and then cache of the result for a bit) in a scenario where certs are thought to be good locally to even make an OCSP request at all. So why were all those Macs making so many OSP requests. It would be interesting to get ISP data to see if there was a traffic spike.

So if Apple had sent a bad update revoking their own certs by mistake then that could cause a lot of Macs to DDOS their OSCPs servers as lots of Mac users launch Apps. So the root cause of the incident in the article would be the extra traffic rather than the servers being bad or under provisioned for expected load or even additional load due to lots of new OS installs of Big Sur.

If that proves to be the case, then you might ask why could they not just make those updates into do nothing updates retrospectively. This would be due to hashing to check updates follow on correctly so changing what is supposed to be immutable data that has already been processed by some systems might cause more issues.

Perhaps people running High Sierra on Nov 12th didn't have slow app launches but did see the problem new installs see (the problem in this thread) for a few hours. An issue like that would be lost in the noise of apps not running or running slow.

I'm going to work back from v145 looking for the issuer hash for WWDRCa to see if the serial for 'Mac OS Application Signing' is in there. This is partly out of stubbornness as if the 2 incidents are related then I'll look like less of an idiot on the other forum.

So to update my theory. Rather than saying that an update was mis-interpreted and caused the Apple certs to be revoked locally by mistake when going up from v42 -> vX -> vY -> v146 all in one go due to being a fresh install, I'm saying that a vX did actually ask to revoke them but perhaps a vY asking to undo that is getting ignored. Obviously if I can't find vX and vY that match my presumptions then I'm just guessing.

Note: If you're interested at 17:45 in this video here https://developer.apple.com/videos/play/wwdc2017/701/ the update mechanism that is involved with this issue is explained at a very high level. Then have a think about what is in the CT Logs and ponder on whether Apple might have been overly trusting them allowing another CA other than themselves to revoke Apple certificates.

Note that I do appreciate that many users are not interested in a root cause analysis and just want their Macs to work.

My strategy here is that if I do actually find anything definitive, then perhaps it can start a dialogue with Apple engineering to see if a patch will be issued or not. Even if the there is no response, which is most likely, it will still be possible to get the information to the right people.

Given that some Macs can't run Mojave and people reinstall their OS for many reasons, locking the users of such macs out of the App Store won't be an experience Apple will want to be in place even if these users are using very old Macs.

The timestamp will be present in the CMS blob signature with an OID of SEC_OID_PKCS9_TIMESTAMP_TOKEN as shown in https://opensource.apple.com/source/Security/Security-57740.1.18/libsecurity_smime/lib/SecCmsBase.h.auto.html

The easiest way to find the an Apple CMS timestamp might be to write a program that uses the open source code from Apple to do it. It may be possible to view the signature with command line tools perhaps after shaving some bytes off.

In either will certainly be possible to grab timestamps of when each update was generated in Apple CMS system of updates to stuff. This way we can say, on time X there was an update which revoked a lot of certificates and on time Y there was an update (v126 in this case) which un revoked them.

Those pieces of data will show that Apple did issues updates to all Macs on High Sierra upwards saying their own certificates were revoked at time X and then not at time Y. I need to check for update X first which I suspect will

be v125.

We must then ask why did Apple do this ? Answers might be.

a) By mistake

b) There was a cyber attack against Apple perhaps which worked by sending poisoned certificate transport logs to them

c) Something else.

Anyhow the incident appears to be very short lived so Apple should be commended for handling it sharply. The only piece of fallout still on going appears to be issue in this thread, is that the error+correction is not handled correctly on new installs of High Sierra.

Anyhow one thing about donor database to note, is that a version at 126 or further from a working Mac is probably a good idea as that will be well past this incident.

Having v146 in all the files may be a red herring or the key to a fix.

Every single update file currently being served as http://valid.apple.com/g3/vX reports inside a version of 146 rather than X.

It could be Apple trying to fast-foward everything to a known starting point and they'll send further updates in due course.

We really can't tell if:

a) This is normal

b) Its not normal and an accident

c) Its not normal but is part of a fix attempt by Apple.

What I think the effect will be is that no matter where you start (today) that your database will leap frog to a version 146 (There are many version 146s as a result). For fresh installs you'll miss those critical deletes I've been talking about and shown in the picture of the plist editor with lots of red text on it.

This does however lead us to a different fix for fresh HighSierra installs without involving a donor DB and just running something once which could be this:

1) sudo sqlite3 /Library/Keychains/crls/valid.sqlite3 'update admin set ival=125 where key = "version" '

2) Reboot

3) Be patient, wait a while (perhaps up to 3 hours)

4) Try to run an app store app and if it does not go back to step 3

5) Wait even longer to see if the fix holds (perhaps leave a machine running over night)

6) See if running an App store app still works.

The machine should be connected to the Internet at all times.

There are tricks to increase the update interval which I've posted before but to keep things simple I've not included them here. I've just done this and am at step 5 now and will report back if it worked.

Running this query below returns no results which is encouraging:

/usr/bin/sqlite3 /Library/Keychains/crls/valid.sqlite3 'SELECT groupid FROM serials where hex(serial) IN ("04CA81F77D5E33F7", "0EEB5787E79E098D")'

In the plist it it holds sha-256 hashes of CA certs to delete so you'd need to delete from the other tables by group id and then do the issuers table last.

462 issuers matched what is found in the seed db at /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3 to the plist from version 126.

Note that one of these (FA8B9F99DBB94E7B772AA9190846E777047C15C7A3BF4A1AF9C0CA984A689511) does appear in the hashes table speaking about it as just a normal cert rather than CA.

I'd double check the Apple source to make sure it does just delete this way before going all in on this solution and also watch out for zero padding vs not perhaps.

The other way, though harder to orchestrate than the DIY method is to move the version so that trustd does all the work for you, but that's not a nice instant fix of course :-(

It may be fun to see which CAs these all are - for example:

https://crt.sh/?q=764A0D84D5552CD5872C73464F37F02175CDA70588102B13ADA2A0199FC403E9

Note: I did get a very key part of how the protocol works significantly wrong.

What is put at http://valid.apple.com/g3/vX is not version X but a complete update to get a machine that from version X right to the current version in one hit. So it's the delta from X to T(oday).

So this explains that it is normal for them all to include version T inside of them and to have the same digests and all be replaced when T moves on. It's also a much more sensible protocol to avoid out of date machines having to cycle through versions, which did strike me as weird but I didn't have the code loaded into an IDE and was just looking at it on the web making jumping between files a pain. Tonight I loaded a later version of XCode into a Catalina VM just to navigate it better and saw my dumb mistake.

So the bug is pretty simple. For a while what was at http://valid.apple.com/g3/v42 would have been bad. New High Sierra installs would be the only Macs that were on 42 as that was the seed (or snapshot) as it's called in the code.

I still have a copy of that from before so looking at the difference between the v42->v146 vs the v42->v147 update will show what's going on.

I hear there is no problem with the Apps if no Internet connection!?

I wonder if they fixed it on Dec 10th?

Possible fix for App Store apps do not install or working properly…

Sec Update -007

https://support.apple.com/kb/DL2026?locale=en_US

@DCamaion: can you compare the Apple-owned Certificates in KeyChain? Compare the ones from when you boot with the HDD, vs the ones you boot from the external drive.

Things I would be looking for:

• Are the certs the same? Any missing? Any extra?
• Are they all in System Roots, or are there some in other containers like System or login or Local Items?

Hey, DCmaion, something is not right. You said this:

Line-by-line the System Roots Apple certificates in the SSD installation look the same as in the HDD installation. Specifically, Apple Root CA, Apple Root CA - 1, Apple Root CA - 2, Apple Root CA - 3, and Apple Root Certificate Authority.

But what I have in mine is

• • Apple Root CA - expires Feb 9, 2035
  • Apple Root CA - G2 - expires April 30, 2039
  • Apple Root CA - G3 - expires April 30, 2039
  • Apple Root Certificate Authority - expires Feb 9, 2025

When you run the codesign with --extract-certificates, this should create files, which would be the CA certificate chain. Either upload here, or make available somewhere else, I will convert them.

Progress made. But may have broken something else, still investigating.

The same issue was reported on 1Password community site, and one suggestion that has been posted actually got MS Remote Desktop app to "magically" start working. I say magically because the cause of things starting to work again is still to be narrowed down.

I installed the "What's Your Sign" app from https://objective-see.com/, so to see if the issue would also get resolved the same way, and it did for me.

1. Install it, follow the prompts.
2. Right click on your app, click "Signing Info"
   1. 1st time, it will show the same "Unavailable" signer info.
   2. Do it a second time, it will now show all 3 CAs!

Once the above has been done, you may be able to launch your app.

WARNING: I don't know what this 3rd party app does, and once I tried to log in to this Apple Discussion site, Safari would not connect anymore to the Apple auth server. I don't know if this is because HS did not have the latest Security Patch (hence maybe supporting stronger TLS than what this site offers), or if this is related to this 3rd party app breaking something else.

I also did notice a new entry in KeyChain being created at around the same time I installed that 3rd party app. The entry name showed "<unknown>", and looking at the details of the entry it was related to com.apple.LaunchServices.TrustedSignature.

I'll have to see if rebuilding the Launch Service Database could be a safer workaround that using this 3rd party app, for which the source code is available. Next in line: source code review...

After leaving my setup alone yesterday, out of frustration of not figuring out the issue, I decided to give it a shot again this morning before going back to work.

All MS Remote Desktop was still failing to launch, no change.

Then I messed around and things started to work again. Here are things that I believe could have an impact on getting things to work again:

1. I still had the two Apple WDRCA certificates manually added to the Systems keychain. I had added them two days ago. No idea on impact, but can't hurt.
2. One of those commands, and some delay afterwards (maybe a minute??) seems to have changed something

spctl --status

spctrl --master-disable

spctrl --add /Applications/Microsoft\ Remote\ Desktop.app

spctrl --assess /Applications/Microsoft\ Remote\ Desktop.app

spctrl --remove /Applications/Microsoft\ Remote\ Desktop.app

spctrl --master-enable

But yet again, the strange thing is that once I can get the app to start, I can no longer log in to the discussions.apple.com communities, as Safari then can't connect to the server I get redirected to for authentication!

The way this is starting to look is that High Sierra builds a CA Trust Chain with one CA or the other, allowing me to start an app (CA1) or allowing me to connect to the auth server (CA2).

Time to reboot, see if some caching gets cleared, and try to reproduce what allowed the app to start.

Possible workaround found. I still have to understand what the 3rd party app "What's Your Sign" does to the OS, but this workaround is now working for me:

1. From a clean reboot, run in terminal "spctl --assess -v /Applications/YourApp.app"
2. From a Finder window, under Applications, right-click (Ctrl-Click) on your app, and select "Signer Info"
3. Enjoy your app

And I confirm that the 2 CA added manually are not required. So please, try the steps above and report if this works for you.