App Store apps do not install properly on a fresh install of High Sierra on internal SSD or HDD

Riverside_Guy wrote:

My current build is 17G14042... my bet is you have yet to do the 006 security update?

Correct—with all that's going on I've been holding off on that. However, my client's computer that originally got me here has had the 006 update and it's still working; that's where I replaced the new broken 17mb database with the old 6.4mb database.

I'm out of my depth here, so forgive me if this is irrelevant. And I don't know if this 0X1 is the same as the flags you guys are talking about. This is from the crash report created when Pages refused to launch. See last quoted line:

Process: Pages [716]

Path: /Applications/Pages.app/Contents/MacOS/Pages

Identifier: com.apple.iWork.Pages

Version: ???

Build Info: Pages-5683000000000000~2

Code Type: X86-64 (Native)

Parent Process: ??? [1]

Responsible: Pages [716]

User ID: 502

PlugIn Path: (null)

PlugIn Identifier: (null)

PlugIn Version: ??? (551.5)

Date/Time: 2020-11-29 09:20:49.892 -0700

OS Version: Mac OS X 10.13.6 (17G14033)

Report Version: 12

Anonymous UUID: 95514DFF-9BAB-1034-E98E-8E1D63E0DC00

Time Awake Since Boot: 310 seconds

System Integrity Protection: enabled

Crashed Thread: 0

Exception Type: EXC_CRASH (Code Signature Invalid)

Exception Codes: 0x0000000000000000, 0x0000000000000000

Exception Note: EXC_CORPSE_NOTIFY

Termination Reason: Namespace CODESIGNING, Code 0x1

Thanks, that's a very helpful data point because it says two things:

1) It ups the probability of a long lurking bug which just didn't get triggered until recently.

2) That deleting /Library/Keychains/crls/valid.sqlite3 is probably a bad idea and using a trusted donor a much better idea.

Note that the donor should likely be a High Sierra Mac such that:

db_version = 5

db_format = 3

What is quite funny about the other forum was that I was saying that I thought that the OS was polling every 3 hours or less to get updates and updating a local database of what is or isn't revoked and that was thought to be wrong. Today we read the code that does just that and can download the updates with a web browser if we want ;-). I like to think I'd have found /Library/Keychains/crls/valid.sqlite3 if I kept at it on my own, but there is no point when others here have done a bang up job already. Thanks once again for the link to this forum. I think now with what everyone's been doing there is more knowledge out there as to how this part of the OS works, which is a good thing to come of the whole debacle.

johnno_uk wrote:

I could be wrong on all this off course but sending an empty update looks suspiciously like the first step in a plan towards fixing at least one category of user, those who install a fresh copy of High Sierra today or later on. Like I say though I think they'll bundle those that did an install before v146 into the same bucket and tell them just to reinstall because it's easier for them to do it that way.

I think I actually understand you here. In other words, they fixed something on the back end, and simply telling people to reinstall their system — because it's easier for them and I actually agree, remember I work with a lot of technically unsophisticated users —will cause whatever that back-end change was to take effect. ("Reinstall the OS" is a knee-jerk response Apple techs have to almost any problem, a long-standing gripe of mine.)

It's probably academic at this point, but if you look at https://discussions.apple.com/thread/252037712?answerId=254019170022#254019170022 where I posted a Finder screenshot of my backup database and the current database, there's a slight size difference in addition to the updated version number. If anybody wants copies of both to compare them in case the information would be helpful, let me know.

I had a similar situation like the ones you describe here (update from Sierra to High Sierra two weeks ago, some apps not running). I chose to follow the 'donor database import' path and it worked all right - the apps are ok again - but still some problems persist. When I try to run Xcode, it wants to install additional components (which I guess is a standard thing when the system was just upgraded), but this installation fails with "an unknown error", which, upon closer inspection (by looking at install.log) turns out to be - guess what - a problem with expiration of some certificate:

Package: PKLeopardPackage <id=com.apple.pkg.MobileDeviceDevelopment, version=10.3.9000000000.1.1488876279, url=file:///Applications/Xcode.app/Contents/Resources/Packages/MobileDeviceDevelopment.pkg> Failed to verify with error: Error Domain=PKInstallErrorDomain Code=102 "The package “MobileDeviceDevelopment.pkg” is untrusted." UserInfo={NSLocalizedDescription=The package “MobileDeviceDevelopment.pkg” is untrusted., NSURL=file:///Applications/Xcode.app/Contents/Resources/Packages/MobileDeviceDevelopment.pkg, PKInstallPackageIdentifier=com.apple.pkg.MobileDeviceDevelopment, NSUnderlyingError=0x7fd134810da0 {Error Domain=NSOSStatusErrorDomain Code=-2147409654 "CSSMERR_TP_CERT_EXPIRED" UserInfo={SecTrustResult=5, PKTrustLevel=PKTrustLevelExpiredCertificate, NSLocalizedFailureReason=CSSMERR_TP_CERT_EXPIRED}}}

any ideas as to what can be done to fix this?

Looking at the bgrep results it appear that it is common practice in every Apple update to say that they're own certificates should not be revoked so rather than a correction to a mistake what we might just be seeing is the normal traffic almost like a proactive correction to a mistake which may have never happened. Updates after v126 don't mention the WWDR cert all before do.

Perhaps locally the trust sort of of times out due to a local time to live of sorts probably not in time per se but in number of update polls whilst online or something like that. So we may never find a vY that said to revoke all the Apple certs after all.

Perhaps all that was wrong here was v126 (and v125 too I've just looked) being mis versioned and no vY where Apple revoked their certs or that somebody infiltrated them to make them do it.

It may all come down to the the wrong version number being put into updates.

Perhaps an engineer hardcoded 146 somewhere whilst testing something and that accidentally made it's way up to Apple's update deployment systems. It could be as simple as that.

It may be the fact that Apple appear to have stopped proactively returning that their own certs are always good that is the issue. Apple was doing until recently pretty much the same thing as what FrancoisQC's automator script was doing although for more certificates.

Due to no timestamps in the updates and no time machine to fetch them from the past, we won't actually ever know if these are what was originally there or if they have perhaps been changed either. I think only Apple will ever know the true root cause of this but I'm leaning less to the theory that I'd find any updates which said to revoke the Apple certs more that what appear to be common practice updates saying that Apple's certs are not revoked got mangled and missed.

johnno, I'm following your adventures pretty closely. It looks like everything you're doing involves the database in /Library/Keychains/crls/valid.sqlite3. If I've been understanding things correctly, this is somehow updated from a corresponding database inside of /System/Library/Security/Certificates.bundle. If I was to use my pre-existing HS install as a donor for my client's fresh HS (I need to try to get her fixed up again today), do you think I should replace both of those, or just the one in Keychains?

I'll go snag both of them from my current system right now. I'm getting ready to apply the 006 security update to my pre-existing HS, and who knows, that might blow up my working system! I have TM and CCC backups also.

I hope you're remembering to eat and sleep ;)

The message from Guillermo316 gave me anothing idea to try out. Install Sierra from scratch, then upgrade to High Sierra using the App Store.

No luck. After forcing a ValidUpdateInterval of 60 seconds, version 42 of valid.sqlite3 got updated to version 146 after about 25 minutes of update running High Sierra.

Pages and Numbers downloaded while at version 42 could launch just fine until the DB got upgraded to version 146.

So word of advice: if you are running Sierra, stick with it for the time being.

And I would have brought it up: the screenshot posted on page 24 where the URL was added as v126 does show a version 146.

I am having the exact same issue. Clean installed High Sierra on a 2010 MBP. Tried with two hard drives -- one brand new SSD, one older one. Any apps downloaded from the App Store will not launch. These include basic Apple software like iMovie and Pages. iMovie was running fine on this machine before we changed the hard drive (the old one got corrupted). Other apps downloaded through the web (Chrome, Microsoft Office, etc.) are working fine.

Apple Support kindly spent a long time on the phone with me, but so far they have not figured out the problem. They are supposed to be looking into it.

In the details of the Problem Report that is created when trying to launch, this comes up -- not sure if it could be relevant:

Crashed Thread: 0

Exception Type: EXC_CRASH (Code Signature Invalid)

Exception Codes: 0x0000000000000000, 0x0000000000000000

Exception Note: EXC_CORPSE_NOTIFY

Termination Reason: Namespace CODESIGNING, Code 0x1

The "delete/rename the file" one lasts mere hours for me, you're automator script that does an edit to the file lasts me 3-4 days. THAT is the temporary fix that I am going with. Maybe a re-post of the download link for all the new folks in here??

One thing I wanted to look into but didn't have time to do: Johnno_uk, are you able to post all those "delete issuer serial numbers" you extracted from the plist here, as text?

It wouldn't be hard to add those other CA serial numbers to the automator script, to make it easy for people here to "double-click to fix" rather than "open a terminal a copy / paste to fix".

In the plist it it holds sha-256 hashes of CA certs to delete so you'd need to delete from the other tables by group id and then do the issuers table last.

462 issuers matched what is found in the seed db at /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3 to the plist from version 126.

Note that one of these (FA8B9F99DBB94E7B772AA9190846E777047C15C7A3BF4A1AF9C0CA984A689511) does appear in the hashes table speaking about it as just a normal cert rather than CA.

I'd double check the Apple source to make sure it does just delete this way before going all in on this solution and also watch out for zero padding vs not perhaps.

The other way, though harder to orchestrate than the DIY method is to move the version so that trustd does all the work for you, but that's not a nice instant fix of course :-(

It may be fun to see which CAs these all are - for example:

https://crt.sh/?q=764A0D84D5552CD5872C73464F37F02175CDA70588102B13ADA2A0199FC403E9

I can see Apple pushing an update soon, that will need a reboot, and that will replace the seed (I like the term!) at /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite with one that Apple know will be able to "accept" and upgrade without any problem what they push from valid.apple.com.

$ shasum /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3
3d2cf91a1d5bc3c20b2dd9b4819f92b1d2a82bb0  /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3
$ ls -lh /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3
-rw-r--r--  1 root  wheel    15M  1 May  2018 /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3

Lets see if this file gets updated in upcoming updates.

And I'm with John_uk on this topic, I think this is the end of it. Apple pushed a fix, everyone is happy. And I'd be very curious to know how much we actually helped Apple fix their issue, how many Apple engineers read what was posted here. The research documented here was quite intensive (and interesting!).

... now all I need is a new GPU for my late 2009 27" iMac !

I found the smoking gun finally.

Not only that, I got this file crawled by Wayback machine I think in the nick of time. I also donated $50 to help pay for that.

https://web.archive.org/web/*/https://valid.apple.com/g3/v42

This means another researcher could independently from me crack open that version knowing it is a copy of what Apple was hosting at the time that I could not have altered and see the same for themselves and also that if Apple muck up this file for future High Sierra new installations there will be an external copy of it.

I had looked at these 6th plists before and seen the WWDR cert in there but not noticed the Valid flag and focusses instead on the 1st plist. If I'd read the source code better I'd have gotten to this sooner.

Occam's razor says it really was that simple after all.

Nice find! We all have been saying that Apple had revoked their cert, now you nailed it!

I'd be curious to know, how many rows shows "No" in the original-and-broken v42, vs how many shows "Yes".

The fact that I was getting TLS error when trying to sign in to this forum (a Revoked error) tells me that they may have marked more than one cert as revoked.