App Store apps do not install properly on a fresh install of High Sierra on internal SSD or HDD

Which is what I suggested in my post in this thread just yesterday but a rare person might come across it. Go check it

https://discussions.apple.com/thread/252037712?answerId=253963482022#253963482022

Not too fast ! Sadly, it only works the time the system is rebuilding the file.

If you run the command

sudo rm -f /Library/Keychains/crls/valid.sqlite3

Without killing trustd, you get en empty file.

-rw-r--r--  1 root  wheel  0 26 nov 16:01 /Library/Keychains/crls/valid.sqlite3

That's why it let apps launch

When you reboot, just wait a few minutes and let the database being filled.

Then no app will launch.

The good command to clear the cache is from https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution/resolving_common_notarization_issues :

sudo killall -9 trustd; sudo rm /Library/Keychains/crls/valid.sqlite3

If you launch one app, trustd restarts and begins to fill the database.

Size of the file when database is full :

-rw-r--r--  1 root  wheel  17293312 26 nov 16:08 /Library/Keychains/crls/valid.sqlite3

When it is full, no app will launch.

So, I am still waiting for a better solution.

By "old" database I mean the one that's broken. I think it was about 17 MB…? I don't have that computer here anymore so I can't check. I replaced that with the one provided by softmusic at https://discussions.apple.com/thread/252037712?answerId=253982289022#253982289022 which is about half the size.

I now see why you asked the question. The smaller database that works is actually the one that predates November 11. The larger broken database is the one that showed up in installs of High Sierra after that. My bad, sorry.

FrancoisQC wrote:

Yes, I preferred blowing one entry away than the entire DB...

"Blowing one entry" is not enough for me because it does not fix every problems I have with the App Store.

For example, trying to reinstall BBEdit that I already owned in the past from App Store is difficult. I have to try it many times.

If I use the old working database, I can reinstall BBEdit without problem. It also fixes other problems from the App Store.

When I use the new database (with or without the serial entry removed)

- There are only 2 incompatible updates listed under "Updates tabs" : Pages and Numbers.

- Many apps show the "Install" button where it should be "Open".

When I use a working database :

- There are 4 incompatible updates listed under "Updates" tabs : Page, Numbers, Garageband and BBEdit

- Apps show "Open" button when they are installed under the "Buyed" tab.

Other ways to get a working database :

1) Quit every apps, launch terminal and type :

sudo killall -9 trustd; sudo sqlite3 /Library/Keychains/crls/valid.sqlite3 'update groups set flags=16 where groupid in (select groupid from issuers where hex(issuer_hash) = "CE057691D730F89CA25E916F7335F4C8A15713DCD273A658C024023F8EB809C2")'

Then reboot.

(This issuer hash should correspond to "Apple Worldwide Developer Relations Certification Authority")

2) Quit every apps, Launch terminal and type :

sudo killall -9 trustd; sudo sqlite3 /Library/Keychains/crls/valid.sqlite3 'delete from serials where groupid in (select groupid from issuers where hex(issuer_hash) = "CE057691D730F89CA25E916F7335F4C8A15713DCD273A658C024023F8EB809C2")'

Then reboot.

3) Quit every apps, disconnect from internet (turn off wifi or remove ethernet cable).

Launch terminal and type :

sudo killall -9 trustd; sudo rm /Library/Keychains/crls/valid.sqlite3

Then reboot.

Reconnect to internet.

Without internet connection, database is version 42 not 145. Flags for the group is 16 whereas 0 in a database version 145.

4) This one is very strange and doesn't work every time. But it explains why some people resolved the problem simply renaming the database file.

Launch terminal and type :

sudo killall -9 trustd; sudo rm /Library/Keychains/crls/valid.sqlite3

Then launch quickly sommes apps or reboot with option to reopen apps before the database is completely rebuilt.

Sometimes it may stops the database from getting filled and will result in an incomplete but working database.

I juste tested this solutions today. They have to be tested for a longer time. I will test the solution 1 from now because it seems to be the safest of the 4.

Deeper understanding of group flags and how trustd works can be made reading the open source code. But I have not enough time and I am not qualified enough to do it. I do not know which version High Sierra use.

Edited, my previous message was clipped... trustd seems to be getting CRLs from

http://crl.apple.com/appleserverauthca1.crl

http://crl.apple.com/root.crl

http://crl.apple.com/codesigning.crl

http://www.apple.com/appleca/root.crl

On Windows, just double-click the CRL file downloaded and you will see the serial numbers that have been revoked. But the thing is, most are empty, or do NOT contain the serial numbers of the certificates revoked (or SN of the issuer).

For me, System Version shows macOS 10.13.6 (17G14042)

Both the 2020-005 and 2020-006 security updates have been installed.

Hello Everyone,

I had this same problem on an install I did back on 23rd November, but wasn't aware of this thread.

I did a fair bit of investigation and can shed some light into what's going on though a lot of that information is already in this thread so sorry if I'm repeating that as I didn't see this thread until today.

The issue is not that the apps are installed incorrectly, they're installed fine I believe.

I say this as I upgraded a system and so many apps were already installed.

The problem is to do with how the OS validates that the apps are good.

In essence it validates two things that are relevant here:

1) Has an app that's on your system been tampered with ?

2) Did you buy it ?

What this means is that there are signatures inside each application that sign it to answer yes to both of those questions.

These signatures are there, but the OS needs to know they are good and it needs some files known as certificates to help it do that. It's a bit more complex than that as there need to be certificates which prove other certificates are good resulting in a chain of them.

a) There is a different certificate used to sign the answer to the 2 questions above.

b) There is a 3rd certificate that proves these first two are good.

c) There is a 4th certificate that proves the 3rd one is good. This is where it all stops as this 4th one is from Apple and will be installed on most computers (Macs or not) all around the world.

There can be more than 4 in total since some apps were signed years ago with older certificates.

People with this problem may well find ancient apps from the Apps Store like one called 'Snap' will still work for example.

In this case though first 3 certificates have not been seen by the new OS installation.

None of them are revoked in the real world or even on anybody's Mac.

The OS is incorrectly reporting them as revoked but that's a red herring I believe but the fix does involve checking that they're not revoked so the jury could still be out there a little bit perhaps.

The trick is to make the OS see and verify them all. Some people in the thread have noted that a tool called WhatsYourSign helps.  Indeed it does but it doesn't cover certificates for receipts (did you buy it).

There is a tool out in the wild called "RB App Checker Lite" which does however.

This fix does not unfortunately appear to be permanent and only lasts for a few hours at least for me.

1. If you start with a fresh install and haven't made too many other changes to try and fix things (especially those that involve Key Chain Access) then installing this app and dragging one a non launching App Store app to it may be all you need to do to fix the problem. The other apps should work to but if one on the off chance is signed differently try dragging it too.

2. If you started with an upgrade from Sierra or lower then step 1 may also be all you'd need to do. However if you've been developing your own Apps in the past then make sure there are no expired copies of the "Apple Worldwide Developer Relations Certification Authority" in your login or system keychain. Having this there was what made some certs show as revoked vs not verified and led me to doing all the other stuff mentioned in this thread like clearing CRL caches etc which I don't think made any difference after all.

Most people can ignore the rest of this post and use RB App Checker Lite, but to do it from the shell something like which is shown below should do the trick I think. I've used an example app that was not working on my machine. Again you only need to do this for one app. I could just post the 2 certs that were involved but they are not mine to post and this is more portable and shows what's going on.

cd /tmp

mkdir fix-mac

cd fix-mac

curl https://developer.apple.com/certificationauthority/AppleWWDRCA.cer --output AppleWWDRCA.cer

codesign -dvvvv --extract-certificates '/Applications/Compare Folders.app'

security verify-cert -r AppleWWDRCA.cer -c codesign0 -R ocsp

# diff AppleWWDRCA.cer codesign1 if you like and they should be the same file - could be clever and just use codesign1

openssl pkcs7 -inform der -in '/Applications/Compare Folders.app/Contents/_MASReceipt/receipt' -print_certs -out receipt-chain.pem

openssl x509 -in receipt-chain.pem -outform der -out receipt.cer

security verify-cert -r AppleWWDRCA.cer -c receipt.cer -R ocsp

Hint: Since the fix is not permanent, you can of course just keep the files AppleWWDRCA.cer, codesign0, and receipt.cer and write a cronjob which runs security verify-cert every few hours in the meanwhile

Note: I had missed the updates from softmusic@ that indeed the CRL database was storing certificates as revoked. This was something I'd looked for independently before finding this thread and had convinced myself, with some help in another forum that although there was an erroneous certificate revocation status, it was not being stored in those particular databases.

I'd also looked at this http://developer.apple.com/certificationauthority/wwdrca.crl (plus the others) as the expected place to find genuine revocations as that would be where I'd expect to find them as that's the authority above the certs being seen as revoked.

softmusic@ appears to be much further ahead with a solution than I am, and it's always best to follow just one person.

Often applying all suggested fixes in scenarios like this can create more mess or confusion.

It stands to reason that getting databases etc into the same/similar state as installations prior to Nov 11 would be a better fix than what' I'm doing which is constantly forcing an online check of status to update them (or caches of them) with correct info, as that creates one thing fighting with another in the system and chaos is never really a good solution to a problem.

From scanning the thread it's hard to tell if somebody has found why certs are being marked locally as revoked. I think the main protocols involved (fetching the big CRL or using OCSP on a cert by cert basis) are all doing things correctly as if they were not (or I missed where somebody here has found this data being received from the net), they'd be knocking out a lot more systems and also the temporary fix of validating signing (which results in an outbound OCSP from trustd when that data is not cached) wouldn't work either.

My suspicions are that all this in general might have something to do with certificate pinning, perhaps to older versions of certificates. Perhaps existing installations re-pinned themselves somehow but new ones get certificates pinned back as if its still 2016 or something really daft like that. When databases start to get new rows in them from out of the blue, then out of the blue will either be online or another source locally. I guess leaving a machine or VM running offline for a long while might help prove things either way. My suspicion is that the smoking gun will be some kind of post install patch, downloaded only once and that the erroneous data used to rebuild the CRL dbs will be sitting on people's machines and getting re-used every now and again.

Looking at where everyone is here with debugging, there is clearly enough info for an engineer at Apple to work with and I suspect they may provide a patch. If not a simple fixing tool could be written an posted on github by the community.

I'm 2 weeks late to the party, and have in effect been having my own fixing party all by myself for a while, so apologies if I've added to confusion here.

Fun fact: Even Apple themselves have suggested clearing this db to fix other issues in the past:

https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087725

That might set peoples minds at rest a little with regard to whether fixes are safe etc.

It also suggests that Apple kind of know that this area of the OS can get into a mess also.

Thanks @johnno_uk, I was not aware of this second signature. After looking at its certs, the leaf certificate "Mac App Store and iTunes Store Receipt Signing" is still indeed valid.

Here is a test that I did. Kill trustd and immediately delete valid.sqlite3. This ensure no memory caching gets written to the file. Then reboot and look at the issuer flags.

Check this:

sudo pkill trustd; sudo rm -f /Library/Keychains/crls/valid.sqlite3 
date;sqlite3 /Library/Keychains/crls/valid.sqlite3 "select s.groupid, hex(serial), flags, format, data from serials s, groups g where hex(serial) in (\"04CA81F77D5E33F7\", \"0EEB5787E79E098D\") and s.groupid = g.groupid "

---- REBOOT HERE and run the next commands right after boot ----

Mon 30 Nov 2020 13:12:24 EST
2587|04CA81F77D5E33F7|16|1|
2587|0EEB5787E79E098D|16|1|

ls -l /Library/Keychains/crls/
total 41072
-rw-r--r--  1 root  wheel   4879220 30 Nov 13:12 update-current
-rw-r--r--  1 root  wheel  16113664 30 Nov 13:12 valid.sqlite3
-rw-r--r--  1 root  wheel     29240 30 Nov 13:12 valid.sqlite3-journal

ls -l /Library/Keychains/crls/
total 41664
-rw-r--r--  1 root  wheel   4879220 30 Nov 13:12 update-current
-rw-r--r--  1 root  wheel  16412672 30 Nov 13:12 valid.sqlite3
-rw-r--r--  1 root  wheel     33344 30 Nov 13:12 valid.sqlite3-journal

ls -l /Library/Keychains/crls/
total 33776
-rw-r--r--  1 root  wheel  17293312 30 Nov 13:13 valid.sqlite3

date;sqlite3 /Library/Keychains/crls/valid.sqlite3 "select s.groupid, hex(serial), flags, format, data from serials s, groups g where hex(serial) in (\"04CA81F77D5E33F7\", \"0EEB5787E79E098D\") and s.groupid = g.groupid "
Mon 30 Nov 2020 13:13:17 EST
2587|04CA81F77D5E33F7|0|1|
2587|0EEB5787E79E098D|0|1|

As you can see, the flag on the issuer certificate (WWDR expiring in Feb 2023) goes initially from 16 (kSecValidInfoAllowlist) to 0 (kSecValidInfoComplete) which means "we have all revocation info for this issuer group".

Adding "Mac App Store and iTunes Store Receipt Signing" to System keychain after the fact will also show that the cert is revoked, which explains why apps will fail to start.

Now what happens during those 53 seconds is left to be explained. The issuer CA only have one CRLdp extension which points to http://crl.apple.com/root.crl, which contains zero certificates revoked and valid dates. So technically even if trustd has "valid CRL info complete" (aka: it has the full CRL), there is no reason to mark issued certificates as revoked.

I split up the binary into its components which was 7 binary plists plus the CMS blob.

Here it is as a zip https://drive.google.com/file/d/1l9eUBWrdCobJyNWLKzRBiLUd_4IZGtkC/view?usp=sharing

I haven't taken a look to see what are in the plists yet and and it's late here.

Hopefully we may see something in here. It wouldn't explain why only new installations are clobbered though unless updating from scratch is broken with this particular update presuming this is the one that is being downloaded which it may not be after all as the latest version is https://valid.apple.com/g3/v146 ...

v42 was the number showing in the system cert bundle so probably was the version that was current when the installer was built or that always ships with High Sierra. I have probably wasted my time splitting that up by hand but it will show what the updates look like anyway.

Anyhow looking at the valid file from another mac that was at version 145 (select * from admin shows this).

I think using a donor database from another Mac really was an excellent idea and it will probably stick.

At least the patient Mac should just follow the same path as the donor.

I now think the problem is probably just that the code in trustd for High Sierra is buggy and it can't upgrade the db from v42 to v145 correctly and makes a mess of it. The other High Sierra Macs in the world will have been doing this version by version.

I also think the essence of such a bug might involve muddling up old versions of intermediate issuer certs with new ones such as the old WWDC cert because in this sense they are both the same issuer. The silly size of the db also suggests an upgrade mess.

Clearing out the database should also work too as this should just build the DB but if the code is buggy, it might also rebuilt it wrong. It doesn't just downlaod the database as a db file though in this instance I don't think. I think it builds them from those compact binaries like https://valid.apple.com/g3/v146. That's still more to go wrong than just taking a donor database to get a machine up to speed.

So I think the best solution was already found and there won't be anything better :-)

johnno_uk wrote:

Clearing out the database should also work too as this should just build the DB but if the code is buggy, it might also rebuilt it wrong. It doesn't just downlaod the database as a db file though in this instance I don't think. I think it builds them from those compact binaries like https://valid.apple.com/g3/v146. That's still more to go wrong than just taking a donor database to get a machine up to speed.

So I think the best solution was already found and there won't be anything better :-)

Thanks johnno

Most of that is completely over my head. I just wanted to make sure you knew that on my pre-existing High Sierra install with the 005 security update applied, if I delete the working /Library/Keychains/crls/valid.sqlite3 (6.4mb) and allow it to rebuild a new one (~17mb), the new one is broken. So I don't think it's something specific to recent High Sierra installers.

You probably understood that already but this is getting to be a really long thread, and it's a lot to expect someone to examine the whole thing.

Johnno_uk was right, the file "/System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3" does get used in rebuilding the file "/Library/Keychains/crls/valid.sqlite3", as when I removed the former (had to boot from USB image) and rebooted, the new file "/Library/Keychains/crls/valid.sqlite3" is definitely messed up. Doing a "select * from admin" does not show a version or db_version, only the "check_again" row even after few hours.

So I guess what we'd need is that someone shares a DB from a working HS that is at version of 145. We can then test if placing the file in both locations, "/System/Library/Security/Certificates.bundle/Contents/Resources/valid.sqlite3" and "/Library/Keychains/crls/valid.sqlite3", will fix the issue.

And Johnno_uk, where did you get those URLs from?

https://valid.apple.com/g3/v42

https://valid.apple.com/g3/v146

The URL path is from https://opensource.apple.com/source/Security/Security-59306.140.5/trust/trustd/SecRevocationNetworking.m.auto.html

It's harder to see in the html version as it mangles the printf format string:

"https://%@/g3/v%ld", host, version

I downloaded the full tarballs of the source as it was easier to work with local files.

It will be worth splitting up v145 into it's plist components to take a look. The original source linked explains the binary format but all you really need to know if that if split by the magic of 'bplist' then the 4 bytes prior say how long that plist is, and just extract that much. The 4 bytes prior to the first bplist and its size marker (e.g 8 bytes away form the first) is how many plists to expect.

My money is still on a software bug and if it was described generically I'd call it a graph node parenting issue. Serial numbers are not unique at all but only unique within the scope of the issuer, so they are in effect meaningless without a link to the issuer. So the DB structure is

dates <- group -> issuers

|

V

serials

I think the terms are little mixed up and really the better way to see it in your head would be to rename things as:

dates <- issuing authority -> issuer CA certs

|

V

serials

That's the only way that would make sense if serials are non unique across issuing authorities (or groups they are called in the schema).

The updates coming down from valid.apple.com appear to prefer using serials + issuing authority to identify the certs rather than a SHA. So I think if we delve into a db copy of v42 we'll find the old WWDC there in a group. Then when it goes to v145 something in v145 may be saying that old issuing CA cert is done and dusted but it's being interpreted incorrectly as to kill off anything from that authority, which would include everything signed off with the newer CA cert.

I'm guessing that older installations will have already done the clearing out of stuff signed with the old cert on a prior update, so there is nothing to do on that score when they get v145 and therefore don't make a mess.