emlynuk

Q: OCSP Service using up quite a bit of bandwidth

I have been tracking down an issue regarding our ISP bandwidth usage (very high).

 

I believe I have found an issue with the OCSP daemon (ocspd) using up quite a bit of bandwidth for no apparent reason - my initial tests seem to show that this daemon, under Mavericks, is using about 100MB of download bandwidth per day (approx 3GB per month).  This is huge considering that this process is meant to cache retrieved results (assuming of course it is getting results).

 

As a further test, I had 2 Macs running Mavericks and 1 running ML overnight, with all machines running RubberNet to monitor per process bandwidth.

On both Mav machines, the ocspd daemon used up the traffic as per above but ML used no bandiwdth for the same process.

 

The implications here is that users with bandwidth limited connections (e.g. Satallite or Mobile) will use up much of their allowance when at idle hence my interest.

 

Can someone verify these findings?

 

Just a wild thought: Perhaps because the keychain is now sent to iCloud in Mav, I wonder if the certificates are being checked more often for security reasons.

 

Thanks

Emlyn

iMac, OS X Mavericks (10.9)

Posted on Nov 10, 2013 5:48 AM

Close

Q: OCSP Service using up quite a bit of bandwidth

  • All replies
  • Helpful answers

Previous Page 2 of 9 last Next
  • by Since 1986,

    Since 1986 Since 1986 Nov 30, 2013 4:31 PM in response to pierrefromsherrington
    Level 1 (5 points)
    Nov 30, 2013 4:31 PM in response to pierrefromsherrington

    For those of you wondering how to achieve this:

     

    "Turning off CRL & OCSP checking in the Certificates Preferences in Keychain Access solved the problem."

     

    Simply set the options to "OFF" in Certificates tab in Keychain Preferences.

  • by Elrainia,

    Elrainia Elrainia Dec 1, 2013 1:33 AM in response to emlynuk
    Level 1 (0 points)
    Dec 1, 2013 1:33 AM in response to emlynuk

    Just wanted to add my name to the list of people with excessive bandwidth usage.  I've upgraded 3 Macs to Mavericks and have the issue on 2 iMacs (one clean install, one upgrade), but no issue on a MBP (upgraded).

     

    I'm seeing approx 1.5Gb per machine per day being downloaded and I think this is up from about 1.3Gb from a month ago.

     

    The Keychain Access preference solution works for me (both OCSP and CRL need to be OFF rather than any other combo).

     

    I have a call into Apple and their supposed to be calling back next week.  Can't say I'm expecting a solution as I'm pretty convinced this is an OS X issue or possibly a server-side issue at Apple.

     

    The only additional info I have to offer is that the downloads don't seem to be as prevelant (or exist at all) when the machine is in use.  I have a tcp-dump running on another machine across ssh; configured to look for packets from Akamai Tech.  The data streams only occur when the machine is idle, i.e. the screen is sleeping, but not the machine.

  • by emlynuk,

    emlynuk emlynuk Dec 1, 2013 1:46 AM in response to Elrainia
    Level 1 (0 points)
    Dec 1, 2013 1:46 AM in response to Elrainia

    Hi Elraina,

     

    Certainly, these requests are being made at any time here, quite often when applications are opened like XCode but also randomly, persumably by background processes.

     

    Personally, I don't think this is a Mavericks issues only as our ML machines are reporting similar traffic patterns.

     

    Would be interested to hear of any response from Apple on this issue.

     

    Emlyn

  • by undertheappletree,

    undertheappletree undertheappletree Dec 1, 2013 1:57 AM in response to undertheappletree
    Level 1 (0 points)
    Dec 1, 2013 1:57 AM in response to undertheappletree

    I have now tried setting OSCP and CRL to "If required", but this appears to have made it worse - back to >1GB a day.  As it's a server there's a possibilty that the addition ocspd traffic was due to users' activity.

     

    For many hours it was trundling along doing a download of ~2.5MB every ~7.5 minutes, then for reasons I've yet to fully understand, the size went up to ~10MB and frequency to ~2.5 minutes.  As per above, it may've been something a user did.  I logged out all bar one iCloud linked user and an unlinked sys admin account (a combination that I know was previously giving me a steady state at 7.5min/2.5MB) and sure enough, it went back to 7.5min intervals, but it's still pulling down the larger volume.  Whatever is going on, it likes multiples of 2.5 minutes, and seems to be synced to clock ie. currently within a few seconds of 00, 7.5, 15, 22.5 .... minutes past the hour.

     

    If you haven't seen it, there's another thread describing some similar issues here https://discussions.apple.com/thread/5606674

  • by Elrainia,

    Elrainia Elrainia Dec 1, 2013 2:43 AM in response to emlynuk
    Level 1 (0 points)
    Dec 1, 2013 2:43 AM in response to emlynuk

    Interesting comment about ML from emlynuk.  I can say hand on heart that I've never seen this pattern on any of our ML hardware.  I only picked up the issue at all because I graph our ethernet ports and on a machine that would normally have minimal traffic overnight, I saw this pattern the first night after my first Mavericks install.

     

    OCSP Bandwidth Pattern.png

     

    Here's a comparative graph from last night with OCSP and CRL turned off (note the Total Out figure on the bottom line of each graph (and the scale)):

     

    Normal Bandwidth.png

     

    I'm not saying it doesn't happen in ML, just saying it's not something I experienced.

     

    I'm also not sure how related it is to apps running.  I'll get an identical graph if I login with no "user facing" apps running.  Obviously there are many processes running in the backgound, but nothing visible on the desktop except for Finder.

     

    I have another completely unhelpful observation to make....  I wonder how many thousands of people have this issue and don't realise it.  If I hadn't got my graph data, the first I would probably have know about it would have been when my ISP contacted me about my bandwidth usage.  It may take a month or two for more people to identify that they have a problem as it could take a while for them to eat though their quotas.

  • by undertheappletree,

    undertheappletree undertheappletree Dec 1, 2013 2:47 AM in response to clockworkapps
    Level 1 (0 points)
    Dec 1, 2013 2:47 AM in response to clockworkapps

    Probably dumb question - my (very limited!) understanding is that OCSP effectively does the same thing as CRL, but uses less bandwidth.  What advantage is there in having both on?

  • by bdiamond18,

    bdiamond18 bdiamond18 Dec 1, 2013 11:23 AM in response to Since 1986
    Level 1 (0 points)
    Dec 1, 2013 11:23 AM in response to Since 1986

    Just wondering if as opposed to completely turning it off (quoting Since 1986):

    >>>>>

    "Turning off CRL & OCSP checking in the Certificates Preferences in Keychain Access solved the problem."

     

    Simply set the options to "OFF" in Certificates tab in Keychain Preferences.

    >>>>>

    If setting the cetrificates to "Require if certificate indicates" would be any better?  At least then there's still SOME checking for certificates (maybe)?

     

    Just changed the settings on my two Macs running Mavericks to see if this will help.  It's been about 10 minutes now, and no more blips.

     

    FYI Apple (if you are listening), this BUG brought my monthly usage to well over my 80GB that I get with my ISP.  I never come close to 80GB, and now I'm going to be paying at least $30 for overage charges!

     

    Thank you to the Apple Discussions Community for having this thread.  I was about to go insane trying to figure out where this was coming from....  I have some lovely screenshots of my router log for the past 24 hours.  Non stop, and this has been going on for days from what I can tell....

  • by Elrainia,

    Elrainia Elrainia Dec 1, 2013 11:32 AM in response to bdiamond18
    Level 1 (0 points)
    Dec 1, 2013 11:32 AM in response to bdiamond18

    > If setting the cetrificates to "Require if certificate indicates" would be any better?  At least then there's

    > still SOME checking for certificates (maybe)?

     

    I hope it helps you, but I'm afraid it didn't appear to make any significant difference on my machines

     

    Let us know how it goes...

  • by stevefrombraddon,

    stevefrombraddon stevefrombraddon Dec 1, 2013 11:54 AM in response to emlynuk
    Level 1 (0 points)
    Dec 1, 2013 11:54 AM in response to emlynuk

    "Turning off CRL & OCSP checking in the Certificates Preferences in Keychain Access solved the problem."

     

    Same for me ... practically down to Zero now. As an earlier poster mused and I agree .... I reckon there are tens of thousands of peeps out there who are not aware their monthly data allowance is being munched up.

     

    I phoned Apple back with my previous reference Job#  [fruitless x 4 calls] and pointed out the shortcomings of their 'help' when the problem and solution was being discussed and identified on their own community board.

     

    I understand their 'engineers are now looking ito it'

  • by bdiamond18,

    bdiamond18 bdiamond18 Dec 1, 2013 11:56 AM in response to Elrainia
    Level 1 (0 points)
    Dec 1, 2013 11:56 AM in response to Elrainia

    Thanks for letting me know, I won't get my hopes up, but I have to try just to see.... so far, so good.

     

    I've downloaded Hands Off! and am monitoring through there (that's how I found the runaway connection in the first place) and through my router logs.... It's been ok so far.  Just one blip, but that might have been my wife going on the computer (not while I'm testing!).

     

    Anybody know how these calls get triggered in the first place?

  • by undertheappletree,

    undertheappletree undertheappletree Dec 1, 2013 12:00 PM in response to undertheappletree
    Level 1 (0 points)
    Dec 1, 2013 12:00 PM in response to undertheappletree

    Logged out all bar sys admin running a network traffic monitor and one iCloud linked account with no apps running overnight.  Downloads were regular at about 40 minute intervals (vs ~7.5 minutes with Mail running in the iCloud linked acct), with no other traffic to speak of, but size of each download appears to have been larger.  Conclusion = Mail, which has a couple of accounts configured for that user, was contributing to frequency of download.

     

    As others have observed, when there's plenty of user activity the regular download pattern appears to be disrupted; happens irregularly.

     

    Has anyone figured out whether the download is a refresh of the whole CRL?  ie. is it downloading a whole new CRL each time?  If so, what's the expiry on the CRL?

  • by Elrainia,

    Elrainia Elrainia Dec 1, 2013 1:30 PM in response to bdiamond18
    Level 1 (0 points)
    Dec 1, 2013 1:30 PM in response to bdiamond18

    > Anybody know how these calls get triggered in the first place?

     

    I don't know whether it's a causal relationship or coincidence, but the first packet of the ocspd managed download always has a corresponing entry in system.log along the lines of:

     

         Dec  1 20:32:53 hostname.deleted.com storeagent[342]: multibyte ANS1 identifiers are not supported

     

    Whilst I have a general understanding of what storeagent does, I don't know enough to draw any meaningful conclusions.

     

    I'm still getting theses events logged regularly, but now OCSP and CRL is turned off, the call to ocspd appears to be being ignored.

  • by Drew Reece,

    Drew Reece Drew Reece Dec 1, 2013 2:12 PM in response to undertheappletree
    Level 5 (7,485 points)
    Notebooks
    Dec 1, 2013 2:12 PM in response to undertheappletree

    undertheappletree wrote:

     

    As others have observed, when there's plenty of user activity the regular download pattern appears to be disrupted; happens irregularly.

     

    It's an 'on demand' job, so it gets put into action via other processes. Because it comes & goes keeping track of the data usage is not possible in Activity Monitor.

     

     

    I couldn't see it ocpsd downloading anything on a clean 10.9 install (Activity Monitor shows the daemon if you enable 'All Processes') the network usage was 0 bytes.

     

    I loaded a site with SSL & it didn't provoke it, I tried the App store (not signed in) it didn't seem to appear.

     

    I setup Mail with my iCloud details & the process started pulling in about 1.5 MB.

     

    I made up a fugly command (that seems impossible to kill via ctrl+c (DO NOT RUN THIS NEXT LINE )

    while TRUE; do echo "Running" ; sudo lsof | grep -i ocspd; sleep 10; done

    It shows the files & connections that have 'oscpd' in the path & loops every 10 seconds.

     

    It looks like it is writing into a few files within /var/root/Library/Caches/ocspd/

    There is a 'fsCachedData' folder but the items don't seem to get new modification times very often.

    The Cache.db-shm and Cache.db-wal do seem to update, with the latter growing over time.

     

    I don't know if that helps anything, just trying to create some ideas on debugging or keeping track of what is going on.

  • by Linc Davis,

    Linc Davis Linc Davis Dec 1, 2013 2:17 PM in response to Drew Reece
    Level 10 (207,926 points)
    Applications
    Dec 1, 2013 2:17 PM in response to Drew Reece

    It looks like it is writing into a few files within /var/root/Library/Caches/ocspd

     

    If you know how to move or delete that directory safely, please do so, then reboot and see whether there's any improvement. Back up all data first.

  • by bdiamond18,

    bdiamond18 bdiamond18 Dec 1, 2013 5:47 PM in response to stevefrombraddon
    Level 1 (0 points)
    Dec 1, 2013 5:47 PM in response to stevefrombraddon

    Went out for a few hours after turning off CRL & OCSP and came back to find the calls were still being made.

     

    *sigh* now I've added a firewall rule to myh router to not allow traffic to these addresses:

     

    devimages.apple.com          canonical name = devimages.apple.com.akadns.net.

    devimages.apple.com.akadns.net          canonical name = a1338.g.akamai.net.

    Name:          a1338.g.akamai.net

    Address: 23.0.165.42

    Name:          a1338.g.akamai.net

    Address: 23.0.165.16

    Name:          a1338.g.akamai.net

    Address: 23.0.165.81

     

    Question is - how do we know when this is fixed?  This is not something I want to run long term....

Previous Page 2 of 9 last Next