symbolic links get corrupted by system process?

moitx wrote:

It's simple to proof the defect: run a software that uses some links on disk for some time (say some hours), then put your Mac into sleep state and then activate it again.

And you guarantee that will reproduce the problem?

My opinion is that Apple MUST resolve this issue quickly. I have spent much (much!!) money for my workstation and this bug is unacceptable. Obviously, the next time I will need a workstation I will remember that.

This thread is almost a year old now. It sure doesn't sound like anything that is causing widespread problems.

This is not a viable way to recreate the problem as it misses what appears to be a major factor. As neaar as I can tell, most if not all of the folks experiencing this problem have a boot drive that is larger than 2TB.

Try this for an alternative approach:

1. install Mac OS X Mountain Lion on a boot drive larger than 2TB

using the entire boot drive as a single large partition.

2. Run the computer for a while. Check for corrupted symlinks from time to time.

Experience seems to show that this will cause corrupted symlinks to appear.

It is possible that there are other aggravating factors which one might need to add to the mix. For example, having XCode installed could be a factor which makes the corrupted symlinks more likely (note that this is a guess based on not a lot of evidence).

I posted a link to a program which might be useful in step 2 above:

https://github.com/daboulet/MacOSX-broken-symlink-finder

How common is this problem? I don't really know although I suspect that it is not all that common considering that boot drives larger than 2TB are probably not all that common.

My personal experience is that this problem started happening within a month of when I first installed Mountain Lion on a brand new Mac Pro (summer 2012 version) into which I had popped four 3TB drives. I have an old Mac Pro (very first version that Apple released) that runs Lion and has had 3TB drives in it for quite a while. It has never seen this problem. This is why I believe that Mountain Lion is a necessary factor.

I reinstalled Mountain Lion from scratch (i.e. reformatted the hard drive) a couple of times on 3TB drives in my new Mac Pro in December of 2012. This problem reappeared in less than 10 days after each reinstall. I then partitioned the drive into a roughly 1.9TB partition and a second roughly 1.1TB partition. The problem has not recurred.

I have also observed that the Mac Pro specs on Apple's website state (http://www.apple.com/macpro/specs.html) quite clearly that a Mac Pro can have up to 8TB of internal storage using a combination of 512M SSD drives, 1TB conventional drives and 2TB conventional drives. I realize that these are the only options that Apple provides on its website although I find it interesting that Apple offers a 3TB option on its iMacs but only 512M SSD, 1TB and 2TB options on its Mac Pros. This leads me to wonder if there is a hardware limitation in the latest Mac Pro which makes this corrupted symlink problem more likely.

The net result of these last two paragraphs is that I suspect that having Mountain Lion in a Mac Pro on a boot partition larger than 2TB could be an important aggravating factor in triggering this issue.

I wonder if this problem happens on new iMacs with 3TB drives.

A couple of questions:

1. Has anyone ever seen this problem on computers other than Mac Pros? If the answer is yes, please briefly describe your system configuration.

2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

3. Has anyone ever seen this problem on computers where their boot partition is less than 2TB? If the answer is yes, please briefly describe your system configuration.

daboulet wrote:

1. Has anyone ever seen this problem on computers other than Mac Pros? If the answer is yes, please briefly describe your system configuration.

2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

3. Has anyone ever seen this problem on computers where their boot partition is less than 2TB? If the answer is yes, please briefly describe your system configuration.

1. Yes. As I've posted earlier I first ran into this on my iMac 7,1 when I upgraded the internal 2TB drive to 4TB.

2. Yes. On both my iMac 7,1 with a single 4 TB boot drive and on my MacPro 3,1 with a 4x1 TB RAID 0 boot volume.

3. No. My iMac 7,1 with a 2 TB boot drive has not exhibited this problem nor has my MacPro 3,1 when I switched to using a SSD as the boot drive and the 4x1 TB RAID 0 volume as /Users. However I don't think this is definitive since there aren't many links in my /Users partition.

daboulet wrote:

A couple of questions:

2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

Yes. RAID5 from 4x2 TB disks, one (boot) partition > 5 TB. MacPro Server Mid 2012.

The Server never sleeps. I installed some uncritical services on the server, and the symlinks get corrupted rarely. When and why? I found nothing to reproduce.

Something I'm thinking of... Is there an unsupported way to install OSX on any partition that isn't HFS+, like, say, ext3? We could try such a nonstandard install and see if the corruption happens or not.

Something I'm thinking of... Is there an unsupported way to install OSX on any partition that isn't HFS+, like, say, ext3? We could try such a nonstandard install and see if the corruption happens or not.

There have been instances of people formatting their drives for "Case Sensitive" HFS+, and installing (or at least restoring a backup) to the case sensitive HFS+ drive, and Mac OS X would not function properly.

An ext3 file system is "Case Sensitive", so at the start this would not work.

In addtion, there are a lot of addtional HFS+ file attributes that ext3 does not support, such as file flags (see man chflags), resource forks, etc...

Yes I confirm that the bug is really simple to reproduce. See for example the following lines, extracted just after a resume from the sleep state:

moon:~ me$ cd ~/Library/Containers/com.apple.mail/Data/Library/Application\ Support/

moon:Application Support me$ ls -l

total 32

lrwxr-xr-x 1 me staff 46 25 Mar 22:48 AddressBook ->

drwx------ 2 me staff 102 14 Nov 09:09 CrashReporter

drwx------ 2 me staff 68 13 Nov 00:29 Mail

lrwxr-xr-x 1 me staff 47 13 Nov 00:29 SyncServices -> ../../../../../Application Support/SyncServices

drwx------@ 2 me staff 68 23 Mar 18:07 mdworker-mail.sb?

As you can see there are two defects: the first is the link with name AddressBook that now is pointing to the local folder (no destination into the ls output); the other is the mdworker-mail.sb. The bug confirmation was obtained putting the pc into the sleep state leaving Mail open. At the resume, Mail stops to working, crashing at each invocation.

The recovery was has follow:

moon:Application Support me$ rm AddressBook

remove AddressBook? y

moon:Application Support me$ ln -sf "../../../../../Application Support/AddressBook" .

moon:Application Support me$ ls -l

total 32

lrwxr-xr-x 1 me staff 46 28 Mar 18:59 AddressBook -> ../../../../../Application Support/AddressBook

drwx------ 2 me staff 102 14 Nov 09:09 CrashReporter

drwx------ 2 me staff 68 13 Nov 00:29 Mail

lrwxr-xr-x 1 me staff 47 13 Nov 00:29 SyncServices -> ../../../../../Application Support/SyncServices

drwx------@ 2 me staff 68 23 Mar 18:07 mdworker-mail.sb?

moon:Application Support me$ rm -rf mdworker-mail.sb^B/

I repeate my opinion: this bug is relative to the filesystem cache and not to hardware problem.

I pretty much agree that this is a pure kernel software problem. The symptoms that I've seen and read about certainly support that theory. That said, there are some nagging doubts in the back of my brain that this could be exacerbated by a drive controller issue.

Could you please describe the system that you are seeing this on?

Has anyone seen this on one of the new iMacs that officially support 3TB drives (ideally a build-to-order machine that came with an Apple installed 3TB drive)?

Also, my Mac Pros literally NEVER sleep. My new summer 2012 Mac Pro has been powered off for a total of maybe an hour since I bought it last summer (my old Mac Pro has probably racked up fewer than a have dozen hours since I purchased it about a year after the Mac Pros first started shipping).

While it is certainly possible that putting one's computer to sleep could aggravate this problem, there is no way that it is a necessary factor.

I'm going to get a bit technical here . . .

For what it is worth, one of the reasons that I am less than convinced that it is a filesystem cache problem is that Mac OS X does not update the atime, ctime or mtime fields in the inodes for symlinks when the links are used (I just conducted a fairly careful experiment to verify this behaviour). This means that there is no reason for the kernel to write to either a symlink's inode or to its link data when the link is being used to access whatever it is referencing. This makes me wonder how any sort of filesystem cache issue would cause this problem. (I could, of course, be missing something in that there could be some part of a symlink's inode that changes and requires that it and/or its link data be written when the link is used in some circumstances; please clue me in if you know more about this).

I will grant that the Mac OS X kernel might write a symlink's inode even though doing so is not strictly necessary but kernel programmers tend to be pretty paranoid about not causing unnecessary disk i/o. Consequently, the fact that it doesn't need to write the inode or or link data (for long links) makes me wonder why using a symlink would cause this problem. Granted, a filesystem cache issue could easily result in almost random corruption of data on one's hard drive. The catch is that we are not seeing random corruption of data, we are seeing corruption of symlinks which tend to follow a pattern - while any symlink could become corrupted, my experience before I repartitiioned my boot drive was that there were a set of symlinks which were far more likely to become corrupted than all the others on my machine.

daboulet wrote:

2. Has anyone ever seen this on a Mac where the Mac OS X boot partition is larger than 2TB? If the answer is yes, please briefly describe your system configuration.

Yes. MacPro1,1 (2006) running Lion (it’s too old to run Mountain Lion), so that rules out the theorised Mountain Lion requirement (plus earlier posters have also mentioned having the problem under Lion). 3TB boot volume (Seagate Barracuda 6Gb/s), which replaced the previous Barracuda 500GB that died about two months ago. The problem only started after I upgraded the drive, and doesn’t exist if I boot from the stock WD 250GB that came with the machine (also running Lion).

I normally put the machine to sleep when not using it, and it may stay that way for a few days before I use it again. The “usual suspects” for symlink corruption on my machine seem to be Macports-installed libz, Microsoft Office frameworks, DiskImages.framework (which kills Disk Utility) and occasionally Java.

3. Has anyone ever seen this problem on computers where their boot partition is less than 2TB? If the answer is yes, please briefly describe your system configuration.

Yes, same machine as above. I’ve tried reducing the size of the boot partition to see if I can make the problem go away, first to 1.5TB, then to 750GB. The problem still exists for me at 750GB 😟, although I think it may be less frequent than it was before. Your symlink checking program is very useful, BTW, thanks!

I’m seriously considering switching back to the old 250GB as the boot volume, but I’ll need to symlink my home directory onto the 3TB drive to do that (it won’t fit on the 250GB any more). Or maybe I’ll buy an SSD to boot from, as the machine’s been feeling a bit laggy ever since I upgraded to Lion. Or maybe I’ll just buy a new iMac 🙂.

Thanks. Interesting on multiple fronts.

- my older Mac Pro 1,1 has had four 3TB drives for quite a while now,

has been running Lion since very shortly after it was released

and has never had the problem happen. Up until I switched to my

new Mac Pro last summer, it got VERY heavy use. It's still running

all the time although doesn't get used in any real sense very often.

- you're seeing it on a 750GB drive. Gack!

daboulet wrote:

- you're seeing it on a 750GB drive. Gack!

… Yeah 😟. But on the plus side it does seem to be only doing it only occasionally now rather than almost continuously like it used to. Last breakage was yesterday, none so far today, touch wood.

nstanger wrote:

Last breakage was yesterday, none so far today, touch wood.

No breakage until today, so that’s five days without an issue. A big improvement! But clearly still not ideal.

OK. So I bought the suggestion (and literally the product) and have been running for the week. General observations:

- FileXRay does not see the change that result in the corruption. It does see the changes that happens when I reset the symlink to the correct link (delete followed by create)

- Symlink corruption appears to be random.It is hitting Microsoft Office 2008 which I don't even use any more. It did not pick up the change to CUDA libraries that I use for Seti@home yet CUDA driver was corrupted and no longer works

- Using the program that provided to check links seems to show that the target of the symlink gets overwritten with data from elsewhere (seems to be HTML or other data rather than completely random bits). Note - checksymlink program fails to pick up some corrupted links (Particularly when Current->A gets changed to Current->i)

So it appears issue is not something that is formally written to disk in way that file system hooks (via fileXRay can pick up). Almost suggests a corruption of disk cache.

Machine was not rebooted and power saver settings are disabled. It has been running all week processing BOINC tasks and using screensaver lock. Machien is MacPro 3.1 with 6Tb partition on RAID drives.

I've seen the problem on 2 separate MacPro system. One was a 3,1 I believe, and the other is a 2012 system. Both were configured with software 10 raid, a total of 3TB on the former, and 4TB on the latter.

The old machine had the problem with 10.7.x, and the new machine has the problem with 10.8.3.

Neither machine suffered data corruption, just link corruption.

Both machines were used as developer machines so they had many symlinks from Hombebrew, my own scripts, etc.

Both machines had other links go bad -- frameworks, mail.

I replaced all 4 of the disks on the new MacPro. Went from Seagates to WD, because I assumed my disks were going bad. Didn't help.

Neither machine sleeps, and maybe 1 reboot a month.

I have no solution, I'm just frustrated.

daboulet, thanks for the tool, I'll give it a shot.