OCSP Service using up quite a bit of bandwidth

Went out for a few hours after turning off CRL & OCSP and came back to find the calls were still being made.

*sigh* now I've added a firewall rule to myh router to not allow traffic to these addresses:

devimages.apple.com canonical name = devimages.apple.com.akadns.net.

devimages.apple.com.akadns.net canonical name = a1338.g.akamai.net.

Name: a1338.g.akamai.net

Address: 23.0.165.42

Name: a1338.g.akamai.net

Address: 23.0.165.16

Name: a1338.g.akamai.net

Address: 23.0.165.81

Question is - how do we know when this is fixed? This is not something I want to run long term....

Feel like I'm beating on a dead horse here, but wanted to throw this out there, as I found it very strange.

On my iMac, OCSPD keeps calling to DEVIMAGES.APPLE.COM

On my Mac Mini, the calls are to www.apple.com

I've turned off checking in the Key prefs, and even created a firewall rule on my router to prevent connectivity to devimages.apple.com. Thought it was strange that one went to a 'dev' IP and the other to a www.

I did…

$ sudo rm -rf /var/root/Library/Caches/ocspd

$ sudo ls -lah /var/root/Library/Caches/ocspd

-->> ls: /var/root/Library/Caches/ocspd: No such file or directory

Rebooted

sudo ls /var/root/Library/Caches/ocspd

-->> ls: /var/root/Library/Caches/ocspd: No such file or directory

I left this for an hour or so (with Mail & App Store open)

The oscpd process appears but the folder remains non-existant, 0 bytes data is shown in Activity Monitor, I toggled the CRL & OSCP settings in Keychain Access. It had no effect.

I recreated the folder & rebooted, the files were created, new fsCacheData appeared. Downloads have started again. It's been running for 30 minutes & the incoming data is sitting at 1.2 MB.

The files in /var/root/Library/Caches/ocspd are now

4.0K 2 Dec 02:08 Cache.db

32K 2 Dec 02:20 Cache.db-shm

443K 2 Dec 02:20 Cache.db-wal

306B 2 Dec 02:20 fsCachedData

My system.log only has my commands that were run via sudo, All messages shows nothing.

I do have the following in an old system.log, possibly after the install completed. It repeats twice.

Oct 29 07:04:35 localhost mDNSResponder[50]: 12: Could not write data to clientPID[83](ocspd) because of error - aborting connection

Oct 29 07:04:35 localhost mDNSResponder[50]: 12: DNSServiceGetAddrInfo v4v6 crl.apple.com. PID[83](ocspd)

Linc, do you know how to log the data a process is sending? Watching Activity Monitor for a process that disapears is frightfully dull 🙂.

I'd be happy dumping numbers into a text file for later calculation if it would help. I'd guess the akamai server names that bdiamond18 is blocking are probably going to change in the future (isn't it a CDN?). Who know what other Apple services that will break too?

I can run wireshark/ tshark if that is the only way to count the data. It's a test setup on a spare disk so it can be erased & reimaged.

I guess it may be a good time to pick one person to erase one of those DB files to see if we can find a specific one to address the problem?

Permissions looked fine on the files before the initial deletion (-rw-r--r-- 1 root wheel), so perhaps the db was in a bad state after install?

It's been running for 30 minutes & the incoming data is sitting at 1.2 MB.

Is that better than before?

Watching Activity Monitor for a process that disapears is frightfully dull

You don't have to. It shows a cumulative total for each process. Mine is about 6 MB over a few days.

Yes it's better, it was a several MB downloaded first time around with about 4.4 MB Cache.db-shm

My ocspd deamon disappears & returns, no log messages suggest a crash or sudden termination. I can only assume it has stopped due to no more requests from Apps. It's an on-demand daemon, so it should spawn at will right?

Any idea on what tasks could keep it running? It returns with 0 bytes in Activity Monitor.

I'll leave it running overnight & compare to a reimaged OS tomorrow if possible.

It runs on demand by registering a Mach service, with WebKit processes as the clients.

@drew

I deleted the contents of /var/root/Libary/Caches/ocspd and rebooted (I didn't delete the actual directory as there didn't see any point). The three files were recreated but NOT the fsCachedData directory.

-rw-r--r-- 1 root wheel 4096 2 Dec 08:40 Cache.db

-rw-r--r-- 1 root wheel 32768 2 Dec 08:40 Cache.db-shm

-rw-r--r-- 1 root wheel 613912 2 Dec 08:42 Cache.db-wal

I renabled the OCSP and CRL options in Keychain Access preferences.... and waited....

Slightly worrying log entry at around the same time:

Dec 2 08:41:14 hostname.deleted.com usernoted[213]: Connection does not have the proper entitlement (com.apple.private.notificationcenter-system) to connect to the system notification center. All communication will be denied. center com.apple.storeagent

The log entry that I always associated with a OCSP download event:

Dec 2 08:41:40 hostname.deleted.com storeagent[353]: multibyte ASN1 identifiers are not supported.

Has not occured since 08:41 when the machine booted.

Two hours in and the files have not been written to since 08:42 (ie. when they were created).

Machine has downloaded a total of 4.49Mb in the two hours (ie. nothing of consequence).

Not sure what my next steps are going to be, but I think I have to look into the log entry above and check that things are working as expected.

Would anyone advise re-creating the fsCacheData directory, or should I just wait and see if the OS does it for itself when it's needed?

Thanks to everyone whose been involved in this discussion, it does feel like we are making some progress...

Update:

Started the Appstore which initated a small certificate download from Akamai.

New file sizes:

-rw-r--r-- 1 root wheel 4096 2 Dec 08:40 Cache.db

-rw-r--r-- 1 root wheel 32768 2 Dec 10:57 Cache.db-shm

-rw-r--r-- 1 root wheel 659232 2 Dec 10:57 Cache.db-wal

And 'lsof | grep ocspd' produced:

ocspd 691 root cwd DIR 1,2 1224 2 /

ocspd 691 root txt REG 1,2 132752 12217 /usr/sbin/ocspd

ocspd 691 root txt REG 1,2 50744 2099199 /private/var/db/mds/system/mdsDirectory.db

ocspd 691 root txt REG 1,2 32768 2099202 /private/var/db/mds/messages/se_SecurityMessages

ocspd 691 root txt REG 1,2 32768 2099255 /private/var/root/Library/Caches/ocspd/Cache.db-shm

ocspd 691 root txt REG 1,2 23548880 48540 /usr/share/icu/icudt51l.dat

ocspd 691 root txt REG 1,2 71616 2100812 /private/var/db/crls/ocspcache.db

ocspd 691 root txt REG 1,2 600832 11337 /usr/lib/dyld

ocspd 691 root txt REG 1,2 343060674 986325 /private/var/db/dyld/dyld_shared_cache_x86_64

ocspd 691 root 0u CHR 3,2 0t0 306 /dev/null

ocspd 691 root 1u CHR 3,2 0t0 306 /dev/null

ocspd 691 root 2u CHR 3,2 0t0 306 /dev/null

ocspd 691 root 3u KQUEUE count=2, state=0x2

ocspd 691 root 4u REG 1,2 4096 2099239 /private/var/root/Library/Caches/ocspd/Cache.db

ocspd 691 root 5u REG 1,2 4096 2099239 /private/var/root/Library/Caches/ocspd/Cache.db

ocspd 691 root 6u REG 1,2 659232 2099254 /private/var/root/Library/Caches/ocspd/Cache.db-wal

ocspd 691 root 7u REG 1,2 32768 2099255 /private/var/root/Library/Caches/ocspd/Cache.db-shm

ocspd 691 root 8u REG 1,2 659232 2099254 /private/var/root/Library/Caches/ocspd/Cache.db-wal

ocspd 691 root 11u unix 0x71451fa84179d6e3 0t0 ->0x71451fa840cb25b3

ocspd 691 root 13u unix 0x71451fa840cb25b3 0t0 ->0x71451fa84179d6e3

ocspd 691 root 85u unix 0x71451fa83cabf1cb 0t0 ->0x71451fa83d5815bb

And now back to waiting....

Please excuse the tiny graph, but it may help somehow. From RubberNet, x axis is time, y download (top) upload (bottom). Time span is a day and a half. You can see very frequent downloads for the first 6-8 hours, then some user activity (me logging users out in the evening). Sparse downloads follow for the night, bit of a blip in the early morning (middle of graph) as I checked what was happening, block of user activity about midday (2/3 of the way along). The user left themselves logged in from then on, so this user doesn't seem responsible for increased download frequency seen at the start of the graph. In fact, they were working solidly for a couple of hours towards the end where there's a notable gap in the traffic.

I am logging them all out bar an unlinked sys admin overnight, will see what it looks like then...

@emlynuk, could you tell if the multiple certificates returned were part of a (or a whole) CRL?

Linc Davis wrote:

It runs on demand by registering a Mach service, with WebKit processes as the clients.

Sorry Linc, that means almost nothing to me.

I took a look at Apple's Developer notes on the kernel & Mach services but that didn't made much sense either.

My oscpd process has changed PID, from 98 to 465 overnight. Is that normal for a Mach service?

It simply means I can't track network usage via the process.

Would anyone advise re-creating the fsCacheData directory, or should I just wait and see if the OS does it for itself when it's needed?

You don't need to recreate it. That happens automatically (or it doesn't.)

Linc Davis wrote:

Would anyone advise re-creating the fsCacheData directory, or should I just wait and see if the OS does it for itself when it's needed?

You don't need to recreate it. That happens automatically (or it doesn't.)

Thanks Linc, that was my conclusion too, but good to hear someone else say it 🙂

Coming up on 6 hours post deletion process and no unwanted downloads so far...

@Elrainia

My fscachedata reappeared after the second reboot. I expected oscpd to recreate it's own folder but it just didn't happen first time around.

It recreated 2nd time with the following permissions…

drwxr-xr-x 9 root wheel 306B 2 Dec 02:20 fsCachedData

I'm afraid I need to figure out tcpdump to get an accurate idea of data usage, I was hoping ps or another command would show network usage per process but that doesn't seem possible to me. I don't think I need Little Snitch & really want to avoid buying it if possible.

@emlynuk,

Do you have a list of urls that seem to be only from oscpd?