Skip navigation

symbolic links get corrupted by system process?

18932 Views 232 Replies Latest reply: Jan 21, 2014 2:54 AM by robertk1 RSS
  • moitx Calculating status...
    Currently Being Moderated
    Mar 27, 2013 11:27 AM (in response to etresoft)

    It's simple to proof the defect: run a software that uses some links on disk for some time (say some hours), then put your Mac into sleep state and then activate it again. That is, it is really possible that you have some broken links. As test software you can use Mail, gcc, Xcode just to give you some examples, fileXray (or hfsdebug) is just a plus.

     

    My opinion is that Apple MUST resolve this issue quickly. I have spent much (much!!) money for my workstation and this bug is unacceptable. Obviously, the next time I will need a workstation I will remember that.

  • etresoft Level 7 Level 7 (23,890 points)
    Currently Being Moderated
    Mar 27, 2013 4:51 PM (in response to moitx)

    moitx wrote:

     

    It's simple to proof the defect: run a software that uses some links on disk for some time (say some hours), then put your Mac into sleep state and then activate it again.

     

    And you guarantee that will reproduce the problem?

     

    My opinion is that Apple MUST resolve this issue quickly. I have spent much (much!!) money for my workstation and this bug is unacceptable. Obviously, the next time I will need a workstation I will remember that.

     

    This thread is almost a year old now. It sure doesn't sound like anything that is causing widespread problems.

  • daboulet Calculating status...
    Currently Being Moderated
    Mar 27, 2013 5:31 PM (in response to moitx)

    This is not a viable way to recreate the problem as it misses what appears to be a major factor.  As neaar as I can tell, most if not all of the folks experiencing this problem have a boot drive that is larger than 2TB.

     

    Try this for an alternative approach:

     

        1. install Mac OS X Mountain Lion on a boot drive larger than 2TB

             using the entire boot drive as a single large partition.

     

        2. Run the computer for a while.  Check for corrupted symlinks from time to time.

     

    Experience seems to show that this will cause corrupted symlinks to appear.

     

    It is possible that there are other aggravating factors which one might need to add to the mix. For example, having XCode installed could be a factor which makes the corrupted symlinks more likely (note that this is a guess based on not a lot of evidence).

     

    I posted a link to a program which might be useful in step 2 above:

     

              https://github.com/daboulet/MacOSX-broken-symlink-finder

     

    How common is this problem? I don't really know although I suspect that it is not all that common considering that boot drives larger than 2TB are probably not all that common.

     

    My personal experience is that this problem started happening within a month of when I first installed Mountain Lion on a brand new Mac Pro (summer 2012 version) into which I had popped four 3TB drives.  I have an old Mac Pro (very first version that Apple released) that runs Lion and has had 3TB drives in it for quite a while.  It has never seen this problem.  This is why I believe that Mountain Lion is a necessary factor.

     

    I reinstalled Mountain Lion from scratch (i.e. reformatted the hard drive) a couple of times on 3TB drives in my new Mac Pro in December of 2012. This problem reappeared in less than 10 days after each reinstall. I then partitioned the drive into a roughly 1.9TB partition and a second roughly 1.1TB partition. The problem has not recurred.

     

    I have also observed that the Mac Pro specs on Apple's website state (http://www.apple.com/macpro/specs.html) quite clearly that a Mac Pro can have up to 8TB of internal storage using a combination of 512M SSD drives, 1TB conventional drives and 2TB conventional drives. I realize that these are the only options that Apple provides on its website although I find it interesting that Apple offers a 3TB option on its iMacs but only 512M SSD, 1TB and 2TB options on its Mac Pros.  This leads me to wonder if there is a hardware limitation in the latest Mac Pro which makes this corrupted symlink problem more likely.

     

    The net result of these last two paragraphs is that I suspect that having Mountain Lion in a Mac Pro on a boot partition larger than 2TB could be an important aggravating factor in triggering this issue.

     

    I wonder if this problem happens on new iMacs with 3TB drives.

  • daboulet Level 1 Level 1 (0 points)
    Currently Being Moderated
    Mar 27, 2013 5:39 PM (in response to ktwalker69)

    A couple of questions:

     

    1. Has anyone ever seen this problem on computers other than Mac Pros? If the answer is yes, please briefly describe your system configuration.

     

    2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

     

    3. Has anyone ever seen this problem on computers where their boot partition is less than 2TB? If the answer is yes, please briefly describe your system configuration.

  • marcosw Calculating status...
    Currently Being Moderated
    Mar 27, 2013 8:55 PM (in response to daboulet)

    daboulet wrote:

     

    1. Has anyone ever seen this problem on computers other than Mac Pros? If the answer is yes, please briefly describe your system configuration.

     

    2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

     

    3. Has anyone ever seen this problem on computers where their boot partition is less than 2TB? If the answer is yes, please briefly describe your system configuration.

    1. Yes.  As I've posted earlier I first ran into this on my iMac 7,1 when I upgraded the internal 2TB drive to 4TB.

     

    2. Yes.  On both my iMac 7,1 with a single 4 TB boot drive and on my MacPro 3,1 with a 4x1 TB RAID 0 boot volume.

     

    3. No.  My iMac 7,1 with a 2 TB boot drive has not exhibited this problem nor has my MacPro 3,1 when I switched to using a SSD as the boot drive and the 4x1 TB RAID 0 volume as /Users.  However I don't think this is definitive since there aren't many links in my /Users partition.

    iMac (24-inch Mid 2007), OS X Mountain Lion (10.8.3)
  • petrahu Level 1 Level 1 (0 points)
    Currently Being Moderated
    Mar 27, 2013 11:26 PM (in response to daboulet)

    daboulet wrote:

     

    A couple of questions:

     

    2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

     

    Yes. RAID5 from 4x2 TB disks, one (boot) partition > 5 TB. MacPro Server Mid 2012.

     

    The Server never sleeps. I installed some uncritical services on the server, and the symlinks get corrupted rarely. When and why? I found nothing to reproduce.

  • Iosepho Level 1 Level 1 (0 points)
    Currently Being Moderated
    Mar 28, 2013 5:47 AM (in response to ktwalker69)

    Something I'm thinking of... Is there an unsupported way to install OSX on any partition that isn't HFS+, like, say, ext3? We could try such a nonstandard install and see if the corruption happens or not.

  • BobHarris Level 6 Level 6 (12,505 points)
    Currently Being Moderated
    Mar 28, 2013 6:24 AM (in response to Iosepho)

    Something I'm thinking of... Is there an unsupported way to install OSX on any partition that isn't HFS+, like, say, ext3? We could try such a nonstandard install and see if the corruption happens or not.

     

    There have been instances of people formatting their drives for "Case Sensitive" HFS+, and installing (or at least restoring a backup) to the case sensitive HFS+ drive, and Mac OS X would not function properly.

     

    An ext3 file system is "Case Sensitive", so at the start this would not work.

     

    In addtion, there are a lot of addtional HFS+ file attributes that ext3 does not support, such as file flags (see man chflags), resource forks, etc...

  • moitx Level 1 Level 1 (0 points)
    Currently Being Moderated
    Mar 28, 2013 11:16 AM (in response to etresoft)

    Yes I confirm that the bug is really simple to reproduce. See for example the following lines, extracted just after a resume from the sleep state:

     

       moon:~ me$ cd ~/Library/Containers/com.apple.mail/Data/Library/Application\ Support/

       moon:Application Support me$ ls -l

       total 32

       lrwxr-xr-x  1 me  staff   46 25 Mar 22:48 AddressBook ->

       drwx------  2 me  staff  102 14 Nov 09:09 CrashReporter

       drwx------  2 me  staff   68 13 Nov 00:29 Mail

       lrwxr-xr-x  1 me  staff   47 13 Nov 00:29 SyncServices -> ../../../../../Application Support/SyncServices

       drwx------@ 2 me  staff   68 23 Mar 18:07 mdworker-mail.sb?

     

    As you can see there are two defects: the first is the link with name AddressBook that now is pointing to the local folder (no destination into the ls output); the other is the mdworker-mail.sb. The bug confirmation was obtained putting the pc into the sleep state leaving Mail open. At the resume, Mail stops to working, crashing at each invocation.

     

    The recovery was has follow:

     

       moon:Application Support me$ rm AddressBook

       remove AddressBook? y

       moon:Application Support me$ ln -sf "../../../../../Application Support/AddressBook" .

       moon:Application Support me$ ls -l

       total 32

       lrwxr-xr-x  1 me  staff   46 28 Mar 18:59 AddressBook -> ../../../../../Application Support/AddressBook

       drwx------  2 me  staff  102 14 Nov 09:09 CrashReporter

       drwx------  2 me  staff   68 13 Nov 00:29 Mail

       lrwxr-xr-x  1 me  staff   47 13 Nov 00:29 SyncServices -> ../../../../../Application Support/SyncServices

       drwx------@ 2 me  staff   68 23 Mar 18:07 mdworker-mail.sb?

       moon:Application Support me$ rm -rf mdworker-mail.sb^B/

     

    I repeate my opinion: this bug is relative to the filesystem cache and not to hardware problem.

  • daboulet Level 1 Level 1 (0 points)
    Currently Being Moderated
    Mar 28, 2013 12:45 PM (in response to moitx)

    I pretty much agree that this is a pure kernel software problem. The symptoms that I've seen and read about certainly support that theory. That said, there are some nagging doubts in the back of my brain that this could be exacerbated by a drive controller issue.

     

    Could you please describe the system that you are seeing this on?

     

    Has anyone seen this on one of the new iMacs that officially support 3TB drives (ideally a build-to-order machine that came with an Apple installed 3TB drive)?

     

    Also, my Mac Pros literally NEVER sleep.  My new summer 2012 Mac Pro has been powered off for a total of maybe an hour since I bought it last summer (my old Mac Pro has probably racked up fewer than a have dozen hours since I purchased it about a year after the Mac Pros first started shipping).

     

    While it is certainly possible that putting one's computer to sleep could aggravate this problem, there is no way that it is a necessary factor.

     

    I'm going to get a bit technical here . . .

     

    For what it is worth, one of the reasons that I am less than convinced that it is a filesystem cache problem is that Mac OS X does not update the atime, ctime or mtime fields in the inodes for symlinks when the links are used (I just conducted a fairly careful experiment to verify this behaviour).  This means that there is no reason for the kernel to write to either a symlink's inode or to its link data when the link is being used to access whatever it is referencing. This makes me wonder how any sort of filesystem cache issue would cause this problem. (I could, of course, be missing something in that there could be some part of a symlink's inode that changes and requires that it and/or its link data be written when the link is used in some circumstances; please clue me in if you know more about this).

     

    I will grant that the Mac OS X kernel might write a symlink's inode even though doing so is not strictly necessary but kernel programmers tend to be pretty paranoid about not causing unnecessary disk i/o. Consequently, the fact that it doesn't need to write the inode or or link data (for long links) makes me wonder why using a symlink would cause this problem.  Granted, a filesystem cache issue could easily result in almost random corruption of data on one's hard drive.  The catch is that we are not seeing random corruption of data, we are seeing corruption of symlinks which tend to follow a pattern - while any symlink could become corrupted, my experience before I repartitiioned my boot drive was that there were a set of symlinks which were far more likely to become corrupted than all the others on my machine.

  • nstanger Calculating status...
    Currently Being Moderated
    Mar 28, 2013 8:23 PM (in response to daboulet)

    daboulet wrote:

     

    2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

     

    Yes. MacPro1,1 (2006) running Lion (it’s too old to run Mountain Lion), so that rules out the theorised Mountain Lion requirement (plus earlier posters have also mentioned having the problem under Lion). 3TB boot volume (Seagate Barracuda 6Gb/s), which replaced the previous Barracuda 500GB that died about two months ago. The problem only started after I upgraded the drive, and doesn’t exist if I boot from the stock WD 250GB that came with the machine (also running Lion).

     

    I normally put the machine to sleep when not using it, and it may stay that way for a few days before I use it again. The “usual suspects” for symlink corruption on my machine seem to be Macports-installed libz, Microsoft Office frameworks, DiskImages.framework (which kills Disk Utility) and occasionally Java.

     

    3. Has anyone ever seen this problem on computers where their boot partition is less than 2TB? If the answer is yes, please briefly describe your system configuration.

     

    Yes, same machine as above. I’ve tried reducing the size of the boot partition to see if I can make the problem go away, first to 1.5TB, then to 750GB. The problem still exists for me at 750GB , although I think it may be less frequent than it was before. Your symlink checking program is very useful, BTW, thanks!

     

    I’m seriously considering switching back to the old 250GB as the boot volume, but I’ll need to symlink my home directory onto the 3TB drive to do that (it won’t fit on the 250GB any more). Or maybe I’ll buy an SSD to boot from, as the machine’s been feeling a bit laggy ever since I upgraded to Lion. Or maybe I’ll just buy a new iMac .

    Mac OS X (10.7.5)
  • daboulet Level 1 Level 1 (0 points)
    Currently Being Moderated
    Mar 28, 2013 10:37 PM (in response to nstanger)

    Thanks.  Interesting on multiple fronts.

     

        - my older Mac Pro 1,1 has had four 3TB drives for quite a while now,

           has been running Lion since very shortly after it was released

           and has never had the problem happen. Up until I switched to my

           new Mac Pro last summer, it got VERY heavy use. It's still running

           all the time although doesn't get used in any real sense very often.

     

        - you're seeing it on a 750GB drive.  Gack!

  • nstanger Level 1 Level 1 (0 points)
    Currently Being Moderated
    Mar 29, 2013 4:12 PM (in response to daboulet)

    daboulet wrote:

     

        - you're seeing it on a 750GB drive.  Gack!

     

    … Yeah . But on the plus side it does seem to be only doing it only occasionally now rather than almost continuously like it used to. Last breakage was yesterday, none so far today, touch wood.

  • nstanger Level 1 Level 1 (0 points)
    Currently Being Moderated
    Apr 2, 2013 3:52 PM (in response to nstanger)

    nstanger wrote:

     

    Last breakage was yesterday, none so far today, touch wood.

     

    No breakage until today, so that’s five days without an issue. A big improvement! But clearly still not ideal.

  • Ed Newman Calculating status...
    Currently Being Moderated
    Apr 5, 2013 7:12 PM (in response to etresoft)

    OK. So I bought the suggestion (and literally the product) and have been running for the week. General observations:

     

    - FileXRay does not see the change that result in the corruption. It does see the changes that happens when I reset the symlink to the correct link (delete followed by create)

    - Symlink corruption appears to be random.It is hitting Microsoft Office 2008 which I don't even use any more. It did not pick up the change to CUDA libraries that I use for Seti@home yet CUDA driver was corrupted and no longer works

    - Using the program that provided to check links seems to show that the target of the symlink gets overwritten with data from elsewhere (seems to be HTML or other data rather than completely random bits). Note - checksymlink program fails to pick up some corrupted links (Particularly when Current->A gets changed to Current->i)

     

    So it appears issue is not something that is formally written to disk in way that file system hooks (via fileXRay can pick up). Almost suggests a corruption of disk cache.

     

    Machine was not rebooted and power saver settings are disabled. It has been running all week processing BOINC tasks and using screensaver lock. Machien is MacPro 3.1 with 6Tb partition on RAID drives.

1 ... 9 10 11 12 13 ... 16 Previous Next

Actions

More Like This

  • Retrieving data ...

Bookmarked By (4)

Legend

  • This solved my question - 10 points
  • This helped me - 5 points
This site contains user submitted content, comments and opinions and is for informational purposes only. Apple disclaims any and all liability for the acts, omissions and conduct of any third parties in connection with or related to your use of the site. All postings and use of the content on this site are subject to the Apple Support Communities Terms of Use.