symbolic links get corrupted by system process?

So we're now just over 2 weeks after a rebuild of our affected MacPro. Our RAID card now has two separate volumes which show up in diskutil list as disk0 and disk1. I think this is significant because someone reported the issue happenning on a large partitioned drive (which would show up like disk0s2, disk0s3). The boot volume is just under 1TB and we haven't seen any corrupted symlinks since.

Certainly not a "fix" but definitely a do-able "workaround." Will report back if I simply didn't wait long enough to see recurrence.

YMMV.

Another million dollar question. Could everyone post whether the corruption happened on factory-installed, Apple certified drives, or on configurations not supported by Apple.

My problems happened on a non-certified 3TB WD drive in a late 2009 iMac. Did anyone experience the problem on, say, a brand new iMac with a factory installed 3TB drive?

My affected client was all Apple factory components.

That's good, that means they won't shrug off your report. Have you set up AIDE? 🙂

I'd love to know the results.

Downloaded, but havent compiled it yet. Looks really cool. We've been checking it every couple days with manual commands. No reports of odd issues from the client either.

My drive is a a single 3TB Hitachi HDS723030ALA640 not originally from Apple. I do not have any RAID configuration.

I have turned off Spotlight and my symlinks have remained intact for a few days now. This seems to reinforce the idea that constant read/write access leads to the corruption.

I noticed that my Spotlight index seems to rebuild a lot. I know there was an issue with the Spotlight index taking a very long time to complete but this was supposedly resolved in 10.8.2, which I am running.

I am hoping for a bug fix in 10.8.3 that doesn't thrash the disk as much. My guess is that the disk is getting a lot of I/O due to some high level bug and this is aggrivating some of the inherant imperfections in HFS+, namely that it tends to become corrupt over time. Or it could be a bug one layer deeper in the logical volume, as suggested by Moitx earlier, that is being aggrivated by the excessive I/O of a higer level bug.

Just guesses.

--

Sam

"and this is aggrivating some of the inherant imperfections in HFS+, namely that it tends to become corrupt over time"

Don't even go there.

There can be no such "inherent imperfection". Becoming corrupt over time is not an "inherent imperfection", it is called a BLOCKING ISSUE. A filesystem that, during normal operation (ie. excluding power outages or magnetic storms), loses data, is not a filesystem.

HFS+ may be outdated and vulnerable, but implying that it, by itself, just by virtue of being used too much, loses information, is like saying that Mercedes cars spontaneously explode if you drive too much in them. (Thankfully they do not, but if they did, you wouldn't see anyone but insane people driving them.)

Apple computers are widely used in data heavy applications, such as movie and music production, newspaper publishing, etc.

If this was an "inherent imperfection", these people would have long switched to Windows. (Which uses NTFS, a filesystem not all that much more advanced than HFS+.)

We'll hopefully soon see what this is. It is not an "inherent imperfection", that is certain.

Hy all,

I have investigated a lot this problem but the solution isn't there (at the moment). What I have found is:

- AIDE seems to confirm that some files were modified during time, also in locations unsuspected;

- I have also enabled the audit daemon, native on OS X, but it has not reported who is the author of the modifications, although was configured to trace all events;

- a strange information coming from the stat command (see a precedent post), as in the following example:

# Broken link

./warnings.py:

Access = Nov 19 21:38:29 2012

Modified = Nov 19 21:38:29 2012

Changed = Nov 22 21:53:42 2012 <--

Birth = Nov 19 21:38:29 2012 <--

Inode = 2454785

# Was:

./warnings.py:

Access = Nov 19 21:38:29 2012

Modified = Nov 19 21:38:29 2012

Changed = Nov 19 21:38:29 2012

Birth = Nov 19 21:38:29 2012

Inode = 2454785

Many other files reports similar informations. If you look with attention, the inode was not modified, that is the file was not moved around into the filesystem, but it's contents is modified. I agree with sam.dorat that Spotlight can be be author of this bug. Another very strange thing is the changed timestamp: it doesn't corresponds to the real time at which the file was modified, as I have seen by the backups (Time Machine) and from AIDE's log.

My machine was assembled by Apple, no modifications at hardware or operating system are in place.

moitx

I have been seeing these symlink corruptions sporadically since sometime this past summer on my mid-2012 Mac Pro. It has three 3TB Seagate ST3000DM001-9YN166 drives (one of which is my boot drive) as well as the came-with-the-computer-when-I-bought-it WDC WD1001FALS-41Y6A1 drive. I have tried different 3TB drives as my boot drive and still get sporadic corrupted symlinks. I just did a from-scratch re-install of Mountain Lion about a week ago in the hopes that that might solve the problem - no such luck as I just ran across a corrupted symlink tonight.

They seem to happen pretty much all over the place although my Canon MX330 printer drivers stopped working due to corrupted symlinks about once every two to three weeks prior to the OS reinstall. By "stopped working", I mean that the app that I typed command-P at would crash and I would soon discover a corrupted symlink down deep in the Canon printer drivers.

I run SuperDuper! every night to backup my boot drive (in addition to Time Machine). SuperDuper! aborts when it encounters one of these corrupted symlinks so I sometimes know that one is corrupted quite soon after knowing that it is ok (I had one case where I happened to have used the symlink just a couple of hours before SuperDuper! ran). As indicated in the last comment above, the inode last change and last modification times did not 'suggest' that anything had actually changed regarding the symlink's inode.

Thanks to motix for the cmdline tips to identify bad symlinks. Only issue to watch out for for those using those cmds is that relative links that are valid will be present in the results since they are relative from the directory that the link exisits in they appear invalid when checked from the root location the commands are executed. Best thing is to save the results to a file and monitor changes via diff on the current and previous file.

Jeff

All,

I've opend bug report 12693117 and recieved a response from Apple requesting I run sysdiagnose and upload the results. Shortly afterwards they closed the ticked as a duplicate and related my ticket to 12133587. From the ticket ID it would appear that the issue has been around for a while. Does the related ticket belong to anyone on the thread? I've asked to be added to the bug ID so I can see the details and the progress but I'm not sure if they will allow that.

Jeff

You can save the results to a file with a simple redirect as follow:

{your_command} 1>/tmp/yourlog.txt 2>&1

If you want to run the commands from a different location (not the directory under check), you can substitute the dot (.) with the name of the directory of interest. A small script that can be usefull to check the same directories at different time is the following:

#!/bin/bash

WATCHED_DIRS="/opt/local /System/Library /Applications /Users/max/Library"

DATE=`date +%Y%m%d%H%M`

LOG_FILE="/tmp/broken_links-${DATE}.log"

echo "Searching for broken links"

for DIR in ${WATCHED_DIRS}; do

echo "Looking into directory ${DIR}"

/usr/bin/find -x ${DIR} -type l ! -execdir test -e '{}' \; -ls | nl 1>${LOG_FILE} 2>&1

done

echo "Done. Your log file is ${LOG_FILE}"

If you change the space separated list of WATCHED_DIRS with the names of your interesting directories, you can collect into a log file the current state of all broken links. You can then use this log to control if some links are gone broken from the last check. To use this script, save it into a file (for example check_broken_links.sh) and give it the execution attribute (chmod a+x check_broken_links.sh)

Just for information, I have disabled all the activities of Spotlight but the problem remains. The commands I have used are the following (to be run as root user):

cd /System/Library/LaunchDaemons/

launchctl unload -w com.apple.metadata.mds.plist

launchctl unload -w com.apple.metadata.mds.scan.plist

launchctl unload -w com.apple.metadata.mds.spindump.plist

You can enable again Spotlight with the same commands substituing option unload with load.

moitx

Just to add a data point to this thread, I've also seen symlink corruption. Experienced hangs on Aperture startup that were traced back to corrupt symlinks in framework directories within application package. Only appears to have started after upgrade to larger disks (6Tb boot drive across two 3Tb drives in JBOD). This is on an older Mac Pro (mid 2008). Software RAID config also had same problem in mirror mode. These are new drives and Mac OSX was completely reinstalled from scratch to try to diagnose Aperture hang.

Would love to solve this as symlink corruption comes back again every week or so. Hasn't been clear whether this was due to Aperture saving state or something else.

Please can someone confirm whether reconfiguring to smaller boot size mitigates problem?