App Store apps do not install properly on a fresh install of High Sierra on internal SSD or HDD

Not exactly what I did, but I believe the results should be the same.

I reformatted the drive using Disk Utility.

I ran the installer from a file in another partition.

I chose no transfer last time, created a new user account, using my Apple ID, and the apps worked.

I’ve reformatted again, clean install, but now transferring over my files from the other partition. It will take several hours to do that. Then I’ll try the apps again.

Reposting what I said in another thread, as this one here seems more alive...

---

Same issue for me. iMac 27, late 2009. Fresh re-install in the last 2 weeks, not sure when. HDD erased prior to reinstall. I had been struggling with the reinstall-from-USB as the first reboot after reinstall kept getting me the grey-progress-bar that never gets to the login screen, which is the reason why I attempted a clean reinstall. I once had issues with the GPU, which I resurrected few years ago by baking it (yes, it worked!!), and booting in safe mode after that clean reinstall allowed me to move forward and kept applying all High Sierra security patches up to 2020-006. Since I could only boot in safe mode and had GPU issues in the past, I tried the trick to move the .kext out so they won't get loaded. Bingo! I could now boot in normal mode (with slow graphics, but that is a start).

Then I ran into the issue discussed here. Only one app downloaded from the App Store: MS Remote Desktop. Same error, because of EXC_CRASH (Code Signature Invalid). ok.

Looking at a MBP running Mohave, 2020-005 not yet installed, I ran into something interesting.

On the MBP, running "codesign -d -vvv /Application/Microsoft\ Remote\ Desktop.app" will show

Authority=Apple Mac OS Application Signing

Authority=Apple Worldwide Developer Relations Certification Authority

Authority=Apple Root CA

But running the same on the faulty High Sierra will show this:

Authority=(unavailable)

If the app can't validate to a trusted CA, that is one possible cause of the code signing error. Moving on, looking at KeyChain on High Sierra, "Apple Root CA" is present. However "Apple Worldwide Developer Relations Certification Authority" is missing, while it is present on Mohave. I was hoping that downloading and installing the missing CA from https://www.apple.com/certificateauthority/ on High Sierra would get things to work again. It didn't. codesign still shows unavailable.

Another odd thing. "Apple Mac OS Application Signing" is not found in KeyChain on Mohave. So how does codesign builds the CA trust chain? Does a .app include signer / CA certificates?

If someone has a working High Sierra setup, I would be curious to look at the codesign output of any app installed from the AppStore, so I can see the Authority being listed.

Do you mean this post? The timestamp doesn't help because of different time zones etc

————————————————————-

FrancoisQC

Nov 24, 2020 11:32 AM in response to scrutinizer82

Reposting, as my typos have confused some people:

1- Do a clean, normal reboot.

2- Get and install What's Your Sign from https://objective-see.com/

Full disclosure, I have no idea why this is required. However the thing it does when checking the signature info seems to have an impact on the next steps

3- In terminal, run

spctl --assess -v /Applications/YourApp.app

4- From a Finder window, under Applications, right-click (Ctrl-Click) on your app, and select "Signer Info"

Thanks @johnno_uk, I was not aware of this second signature. After looking at its certs, the leaf certificate "Mac App Store and iTunes Store Receipt Signing" is still indeed valid.

Here is a test that I did. Kill trustd and immediately delete valid.sqlite3. This ensure no memory caching gets written to the file. Then reboot and look at the issuer flags.

Check this:

sudo pkill trustd; sudo rm -f /Library/Keychains/crls/valid.sqlite3 
date;sqlite3 /Library/Keychains/crls/valid.sqlite3 "select s.groupid, hex(serial), flags, format, data from serials s, groups g where hex(serial) in (\"04CA81F77D5E33F7\", \"0EEB5787E79E098D\") and s.groupid = g.groupid "

---- REBOOT HERE and run the next commands right after boot ----

Mon 30 Nov 2020 13:12:24 EST
2587|04CA81F77D5E33F7|16|1|
2587|0EEB5787E79E098D|16|1|

ls -l /Library/Keychains/crls/
total 41072
-rw-r--r--  1 root  wheel   4879220 30 Nov 13:12 update-current
-rw-r--r--  1 root  wheel  16113664 30 Nov 13:12 valid.sqlite3
-rw-r--r--  1 root  wheel     29240 30 Nov 13:12 valid.sqlite3-journal

ls -l /Library/Keychains/crls/
total 41664
-rw-r--r--  1 root  wheel   4879220 30 Nov 13:12 update-current
-rw-r--r--  1 root  wheel  16412672 30 Nov 13:12 valid.sqlite3
-rw-r--r--  1 root  wheel     33344 30 Nov 13:12 valid.sqlite3-journal

ls -l /Library/Keychains/crls/
total 33776
-rw-r--r--  1 root  wheel  17293312 30 Nov 13:13 valid.sqlite3

date;sqlite3 /Library/Keychains/crls/valid.sqlite3 "select s.groupid, hex(serial), flags, format, data from serials s, groups g where hex(serial) in (\"04CA81F77D5E33F7\", \"0EEB5787E79E098D\") and s.groupid = g.groupid "
Mon 30 Nov 2020 13:13:17 EST
2587|04CA81F77D5E33F7|0|1|
2587|0EEB5787E79E098D|0|1|

As you can see, the flag on the issuer certificate (WWDR expiring in Feb 2023) goes initially from 16 (kSecValidInfoAllowlist) to 0 (kSecValidInfoComplete) which means "we have all revocation info for this issuer group".

Adding "Mac App Store and iTunes Store Receipt Signing" to System keychain after the fact will also show that the cert is revoked, which explains why apps will fail to start.

Now what happens during those 53 seconds is left to be explained. The issuer CA only have one CRLdp extension which points to http://crl.apple.com/root.crl, which contains zero certificates revoked and valid dates. So technically even if trustd has "valid CRL info complete" (aka: it has the full CRL), there is no reason to mark issued certificates as revoked.

Riverside_Guy wrote:

He had me do a safe boot BUT was not able to answer the question about startup items loading (by this point, I doubt he was tier 3, AND he seemed to have no intention of escalating me there) on safe boot, just that it takes it's time because it runs first aid before loading. My mistake was not first deleting the edited db file before running this process, BUT my expectation was it wouldn't have made any difference.

Looks like we have 2 options here... using the automator script to edit the db or installing an older version of it. Frankly I am not so sure the "older version" is as long lasting as some say, there may be events that cause it to be replaced in some fashion. THAT also might be the cause for the "edit the file" approach would fail. Frankly, I am sticking to the "edit the file" approach. It's been 2 days and several boots and it is still "working." We MAY get a Xmas present and have the mothership FIX the issue THEY created, but I have zero hope they will ever do it...

When I first ran into this problem on a client's computer, one of the first things I did was to boot into safe mode and try to run Pages from there; it still crashed on launch. I fiddled around a bunch in safe mode and deleted some caches and preferences — the usual suspects in both library folders — and actually got the App Store to let me download the iWork apps which it would not do before. But after a while that broke again.

I've been a Mac consultant since the late 80s. One thing that has always bothered me about apple's tech support is that the first thing they will tell someone to do is reinstall their system. That's hard to do for a lot of people (a lot of my clients are elderly ladies* and other unsophisticated types) and often causes more problems than it solves. Sometimes it just feels like a convenient way to break off the phone call.

Using the very latest system is usually OK, and unavoidable if you buy new hardware or like to surf the bleeding edge. I tend to stay a system or three behind, letting others discover the pitfalls in the new stuff. Right now I'm trying to nurse along some old software until I don't need it anymore, especially QuickBooks. I've had High Sierra since July and, as described before, have the small 6.4mb database in /Library/Keychains/crls/. Things work fine. I replaced my client's broken database with a copy of mine and things are working for her now too. That's the solution I'm gonna stick with unless and until Apple does something on their end. A workaround that must be periodically reapplied is fine on one's own computer, but impractical when you'd have to travel to somebody else's house to do it to theirs. I haven't installed the 006 security update yet but my client did and it did not appear to break anything.

At any rate, I'm getting ready to upgrade to Mojave because Dictation in High Sierra is abysmal and hopefully it will be better in Mojave. But I still need to follow this issue because, especially in today's economic climate, a lot of people simply can't afford to upgrade their hardware and software.

I've been told elsewhere that High Sierra is obsolete and unsupported, but have not been able to confirm this via any Apple documentation. Does anyone know of a good source to confirm or refute?

------------------------------------------------------------

* it's OK for me to say this because I are one.

In this case its likely not the same 0x1 and probably all that's interesting about that 0x1 is that it's not 0x0 which would probably be good.

What we've been able to see from all the code is that there is a mechanism in the OS, which I think started with High Sierra called 'ValidUpdate' which updates a little database to say if certificates are valid or not.

Rather than updating that database to say some crucial Apple certificates are valid it's doing the opposite and saying they are invalid/revoked.

It's probably not a bug in the code (though sometimes bugs don't kick in due to a date or only with certain data) but rather a data bug. Such bad data could come from 3 places in general.

1) It was in the install image for High Sierra - Unlikely as the current image has been around a long while, though data might have time limits as part of its structure, so not totally out of the question.

2) It was in a post install patch which is like a supplement to the OS install image downloaded from the Internet towards the end of the process

3) It's data downloaded from Apple servers

Assuming its not a code bug, what we're looking for is the smoking gun piece of source data that is wrong and causing all the problems. Basically tracking back further to where the data in a known broken database came from in the first place. With that it will be possible to tell Apple what is wrong and have a good guess at what the fix would be, which if we're really lucky might just be Apple changing some files on a load of web servers (CMS) around the world and then all the broken High Sierra Macs will just correct themselves and it will be like nothing had ever happened.

Along the way though its possible to see more clearly how things work like what the preference options are to turn the feature on or off or make it run more quickly or even have it use a local update sever (which would likely be for enterprise installations).

Anyhow looking at the valid file from another mac that was at version 145 (select * from admin shows this).

I think using a donor database from another Mac really was an excellent idea and it will probably stick.

At least the patient Mac should just follow the same path as the donor.

I now think the problem is probably just that the code in trustd for High Sierra is buggy and it can't upgrade the db from v42 to v145 correctly and makes a mess of it. The other High Sierra Macs in the world will have been doing this version by version.

I also think the essence of such a bug might involve muddling up old versions of intermediate issuer certs with new ones such as the old WWDC cert because in this sense they are both the same issuer. The silly size of the db also suggests an upgrade mess.

Clearing out the database should also work too as this should just build the DB but if the code is buggy, it might also rebuilt it wrong. It doesn't just downlaod the database as a db file though in this instance I don't think. I think it builds them from those compact binaries like https://valid.apple.com/g3/v146. That's still more to go wrong than just taking a donor database to get a machine up to speed.

So I think the best solution was already found and there won't be anything better :-)

The database shared from softmusic back on November 26 is the one that worked for me. The computer I was working on needed iWork apps. Not only did it allow them to launch, it allowed the App Store to download the proper version. https://discussions.apple.com/thread/252037712?answerId=253981944022#253981944022

http://www.mediafire.com/file/m2ky3mrnon6jy49/valid.sqlite3.zip/file

So I just saw what softmusic@ did. I started with v42 and ended up with a v146 that was large and broken.

Which will be pretty much just v145 with a version bump and not v42 bumped. It would be weird for a system of incremental updates to not go back to the last full one, but I can't see why Apple put out v146 if it does nothing. It must do something to some system, or they've just made a mistake.

Below is the script I used to reset it back to look like a fresh install as much as I think might be needed. I had update for an instant update but a reboot did the trick, so no waiting for too long.

Catching the update in the act looked like it did for others:

#/usr/bin/sh

killall -9 trustd

defaults write /Library/Preferences/com.apple.security.plist ValidUpdateInterval -int 60

cp /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3 /Library/Keychains/crls/valid.sqlite3

sqlite3 /Library/Keychains/crls/valid.sqlite3 "delete from admin where key = 'check_again'"

touch -r /System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3 /Library/Keychains/crls/valid.sqlite3

launchctl start com.apple.trustd

For the Xcode issue double check to see if these serials are in the database and if so perhaps just try deleting them.

I got those from the same package path on my machine which has Version 10.1 (10B61) of XCode on it which I think is the latest one which official supports High Sierra.

5AA247A11690DC282EFCF72C5F427DC2

309BF1DB5506323A7853632F556B5F7A

sqlite3 /Library/Keychains/crls/valid.sqlite3 "SELECT groupid from serials where hex(serial) IN ('5AA247A11690DC282EFCF72C5F427DC2','309BF1DB5506323A7853632F556B5F7A')"

If something shows up it would be worth reporting that back up here.

sudo sqlite3 /Library/Keychains/crls/valid.sqlite3 "DELETE from serials where hex(serial) IN ('5AA247A11690DC282EFCF72C5F427DC2','309BF1DB5506323A7853632F556B5F7A')"

Another thing to check for is that you don't have older expired versions of those certificates in your keychains as that can cause issues too which on an upgrade might be the case. I had old copies for the WWDRCA cert from ages ago when I upgraded from Sierra to High Sierra.

jlhopes22 wrote:

1. If I've replaced the valid.sqlite3 file with the older version (as posted by softmusic on p.13 of this thread), is this just temporary -- will it go back to the way it was at some point? Also, will it cause any other problems on the computer?

3. So far I haven't installed the Security Updates from the App Store. With this solution, does it matter if I install those or not?

I did a new HS install on a client's computer and got the problem. I transplanted softmusic's donor database which fixed things. However, when my client installed either both security updates or just 006, the problem reappeared. So I just did the transplant again. We'll see if it sticks, now that there are no more security updates to apply, and if it doesn't stick it will mean that with the passage of time, some sort of automatic update process will break it again.

I can confirm that killing trustd (to prevent cache write to disk), deleting the DB, and rebooting does the trick as of today.

So, the steps

• Utilities --> Terminal, and run those commands

sudo killall -9 trustd
sudo rm -f /Library/Keychains/crls/valid.sqlite3

• Reboot
• Wait 5-10 minutes for the new version to be installed.

To confirm, after reboot, run this:

sqlite3 /Library/Keychains/crls/valid.sqlite3 "select * from admin"

db_version|5|
db_format|3|
db_source|0|valid.apple.com
version|147|
check_again|0|o??????A

version show show 147.

Hopefully this is the end of this saga.

Yes that's exactly what they are up to. They change all the updates including those from the past.

I took a full set of what they had when the end version was 146 and also tried to get a few indexed on the Wayback machine to prove this without it having to be my say so.

The difference this time is the digests are different. I haven't taken a full set yet but I think all the digests will be the same across the updates. I also need to see if the version number is included in the digest hash or not, the code shows this but I haven't looked at it yet.

What Apple may be trying to do is to reset every Mac to it's seed database and are cycling through digests that match the seeds of the different databases that come with different macOS version installations. We might see a new empty update with a different digest let's say every 2 weeks (to give time for machines to see them), cycling through different seed db digests.

Perhaps they were working their way back through the OS versions but have figured that going forward from High Sierra is a better approach given how High Sierra is susceptible to this problem here. I don't think the seed dbs change between point versions of the OS but I haven't looked as I don't have any old install images other than the final ones of each non current major version.

The problem with this theory is they could perhaps do this by sending a deliberately bad update which might be easier.

I don't have the full protocol including digest checks, database reseeding etc mapped out in my head yet as I've only really done a half-baked reading of the source code. Though it's all in there and possible to figure it out.

My other theory from yesterday was that non incremental updates have always been broken and the update generation code was applying these in a broken way to a database server side used only for this purpose (so not breaking the host machine in any way), and then sending out digests for the broken database. They may have fixed full updates at that end and so now the digests will be that of a correct database rather than a broken one. Though if all the digests are the same across increments that would be weird unless increments did nothing. This is why my replaying seed db digests every couple of weeks came to me as an idea of what might be going on this morning whilst in the shower.

Whatever they are up to I'm convinced they are trying to clean up a mess that is bigger than the problem here, but doing so without patching any code but rather being a little liberal with their own protocol to get to the end point they want.

Note that another way to reset that I spotted from the source code.

1. sudo touch /Library/Keychains/crls/.valid_replace
2. reboot

This will make trustd reseed the database for you.

It does look though that for the issue here in this thread that Apple have done what they needed to do. So further investigation will just be to get hints of what the bigger picture might have been and also just to understand the mechanism out of interest. Given what it is, this mechanism is quite important to the security of many computing devices in the world and really there should be people outside of Apple that understand it IMHO.