Q: symbolic links get corrupted by system process?

hstimer wrote:

 

I don't understand "Anyone who has made custom, low-level changes to their operating system is automatically unable to demonstrate said proof regardless of evidence."

I was referring to the corrupted symbolic link in /System/Hidden/Versions/Current. Something unusual was going on there long before any problems occured.

Pretty much anything can decide to write over random places on disk, depending on what you are running. You can't claim it is a kernel or operating system bug if there are any kernel extensions or similar low-level modifications.

This whole thread is very fishy. If anything like this happened to me there is no way on earth I would continue to run and put my data in jeopardy.

This is what people are saying here: "Yeah, some kind of strange corruption is happening on my system. Symbolic links keep getting overwritten with garbage. But hey, I just fix the links and keep on truckin'!"

Really? I mean, really?

hstimer wrote:

 

I wish Apple didn't give up on ZFS. There is a checksum at the start of each block which helps you to detect errors right away.

Don't wait for Apple. If you are convinced that your system has kernel bugs that are corrupting your data and those machines are still powered on, may as well enjoy the ride and be a ZFS power user.

I've rebuilt 2 different MacPro systems from scratch, and in both cases replaced a complete set of 4 drives. Yes, that is 16 drives in total. So yes my first response was to drop what I was doing and exchange in new media, install a fresh os, and reinstall all the apps from scratch. The only file transferring was for files in my home directory.

However, the drives have never tested bad, and the problem keeps coming back, and the only files I can confirm corrupted are soft link files.

Soft link files can not easily be overwritten. Short of writing a buggy kext, or an application that opens drives in raw mode, I'm not sure how it could be done.

I could be wrong, but I'm assuming that none of the applications from the app store can do either of those things. That leaves drivers, os, and around 10 command line apps that I grab through Homebrew.

I use Homebrew specifically because it doesn't need sudo, and last I checked you can't open a raw device unless your are root. There are only a few apps that I run with privileges: wireshark, lsof, vmware, and virtualbox. The latter two install kext, so they could be causing the problem.

I think it is likely that something about how I need to configure my machines is triggering the problem, and that I've got to narrow it down so that I can get someone to fix their bug. I'm thinking it is Apple, but until I can make it reproducible, I can't know for sure.

My prior question still stands; if osx has a microkernel, can subsystems stomp on each other?

hstimer wrote:

 

I use Homebrew specifically because it doesn't need sudo, and last I checked you can't open a raw device unless your are root. There are only a few apps that I run with privileges: wireshark, lsof, vmware, and virtualbox. The latter two install kext, so they could be causing the problem.

 

 

Well on my MacPro that exhibited this problem, I too used Homebrew (avoiding the need for sudo) and did NOT run any of the apps you mentioned (wireshark, lsof, vmware and virtualbox).  Also have tried replacing drives & fully rebuilding multiple times, to no avail.

If you have any suggestions to help try track down the issue......     Data does not appear to get corrupted here so issue is more an annoyance than a data loss risk (at least at the moment)

fileXray on my machine shows that the AttributeModDate is updated for affected files but no other date field (thus file does not appear to show as changed for ls -l). Many of the symlinks appear to be in consecutive node records in the Catalog File Thread Record even though they may not be in same directory folder structure. Corruption appears to have occurred around a 5 minute period

ZFS works great on FreeBSD, but Apple never finished the job of Appleizing it. Last I checked, MacZFS is still case sensitive, and you can't boot from it.

"Many of the symlinks appear to be in consecutive node records in the Catalog File Thread Record even though they may not be in same directory folder structure."

That is very interesting. Can you run "kextstat | grep -v apple" in the terminal and report back the results?

A "df" would be nice too.

Apple has shipped tens of millions of machines with Lion and Mountain Lion. You can post messages here in this discussion forum until you wear your fingers to the bone and it won't do you any good. You need proof. Documentation. Anything, really.

Things are are not going to help your case:

Homebrew

Wireshark

lsof

VMWare

VirtualBox

Both VMWare and VirtualBox on the same system

Anything in /usr/local

Anything at /System/Hidden/Versions/Current

And anything, really, that isn't made by Apple. The thing is, you can't just judge something by whether you think you are avoiding the need for sudo. If you ever give any installer your admin password, you are giving it permission to run anything via sudo anytime it wants to. No password required.

It is unlikely that any of that 3rd party software is causing any problem. Lots of people run all of that with no issues. That doesn't apply to you or anyone claiming this issue. This is an extraordinary claim that demands proof. If anyone has such proof, or even 5 minutes to spare, they should be sending a bug report to Apple instead of bickering with me.

Actually, I thought that was what this thread was for: finding a work around and finding a scenario that is reproducible. That's why you are here, right?

etresoft wrote:

 

Things are are not going to help your case:

Homebrew

Wireshark

lsof

VMWare

VirtualBox

Both VMWare and VirtualBox on the same system

Anything in /usr/local

Anything at /System/Hidden/Versions/Current

As mentioned before, Homebrew is specifically engineered to run with standard user permissions (I.e. NOT requiring sudo).  Since link corruption has been reported in areas such as /System/Library/Frameworks, areas where mere mortals normally don't have license to muck about in, this sounds highly unlikely that Hiomebrew would be the cause.

And lsof is part of the standard UNIX tools that Apple distributes as part of the base OS X install (it is located in /usr/sbin), so if that were the culprit then fixing it is part of their purview.

And there are users who have reported this issue that don't run any of the software mentioned (including myself - I do not use Wireshark, Vmware, etc.)  Not to mention people who have reported this problem as occurring even on "vanilla" rebuilds.

We're just regular users like yourself (except that we happen to know more than most people about Macs). Finding a workaround or a reproducible scenario is a fine goal, but there are 13 pages in this thread with only a few on target.  Most of it is griping about some error people think Apple has made, even though they can't say precisely what the error is.

You all can gripe here all you like, and all you're going to manage to do is make people grumpy.  We're not Apple.

To be honest, I gave up trying to help on this thread months ago, when it became clear to me that people didn't want to solve the problem, they wanted to complain until someone else solved it.  There's nothing I can do with that, since it's not a problem I can reproduce myself.  If you have an earnest interest in solving this problem, I suggest you file a bug report, and if you want to discuss it some more here use Apple's bug report template to describe the problem.  Then we might get somewhere.

I've been trying to track this down for the last few months. I have put in a bug report which Apple kindly responded with "Please go see your nearest Genius bar to check on hardware" and eventually closed....(sigh). Evidence so far appears to be:

- Only appears to happen to people with Boot drives greater than approximately 2Tb. Reducing appears to either make problem go away or reduce chances of occurring. Analysis: guess that majority of users do not upgrade their hardware and thus fall below 2Tb boot partition and thus never see issue

- Symlink corruption appears to only occur to execution code - personal experience suggests things like /Applications though I have seen links in /usr/local/cuda/lib (used to run CUDA applications under BOINC) fail. For me the majority of failures appears to be "Resources" or "Current" symlinks under apps like MS Office. Not seen data failures as yet or corruption to data symlinks (use Aperture which is heavily dependent on symlinks so far with no issue).

- fileXray appears to not log changes to file system even though in monitor mode and symlink failures look like something is writing to file attributes not content (only AttributeModDate is updated when corruption occurs)

- (brief) analysis of latest failure suggests that failures are closely related in Catalog File Thread Table (whatever that is... ;-( ) in HFS+ suggesting some process overwriting same critical data portions

- Corruption is, as yet, unreproducable but appears to occur in batches - i.e. machine works fine for period of time then suffers symlink corruption. Can't find logs to idetify what be running around same time.

There appear to be several people with this issue and all appear to be in the camp of "Upgraded machine to use larger capacity disks than is generally supplied by Apple"

Can you run "kextstat | grep -v apple" in the terminal and report back the results?

A "df" would be nice too.

Ed Newman wrote:

 

I've been trying to track this down for the last few months. I have put in a bug report which Apple kindly responded with "Please go see your nearest Genius bar to check on hardware" and eventually closed....(sigh). Evidence so far appears to be:

 

- Only appears to happen to people with Boot drives greater than approximately 2Tb. Reducing appears to either make problem go away or reduce chances of occurring. Analysis: guess that majority of users do not upgrade their hardware and thus fall below 2Tb boot partition and thus never see issue

- Symlink corruption appears to only occur to execution code - personal experience suggests things like /Applications though I have seen links in /usr/local/cuda/lib (used to run CUDA applications under BOINC) fail. For me the majority of failures appears to be "Resources" or "Current" symlinks under apps like MS Office. Not seen data failures as yet or corruption to data symlinks (use Aperture which is heavily dependent on symlinks so far with no issue).

- fileXray appears to not log changes to file system even though in monitor mode and symlink failures look like something is writing to file attributes not content (only AttributeModDate is updated when corruption occurs)

- (brief) analysis of latest failure suggests that failures are closely related in Catalog File Thread Table (whatever that is... ;-( ) in HFS+ suggesting some process overwriting same critical data portions

- Corruption is, as yet, unreproducable but appears to occur in batches - i.e. machine works fine for period of time then suffers symlink corruption. Can't find logs to idetify what be running around same time.

 

There appear to be several people with this issue and all appear to be in the camp of "Upgraded machine to use larger capacity disks than is generally supplied by Apple"

I'd like to add to this.  In my experience the corruption occurs to apps that are used frequently, and also that (at least for me) corruption wasn't limited to executable code.  For example, I frequently got corrupted links in the Mail.app sandbox container (~/Library/Containers/com.apple.mail), which contains data, not executable code, and Mail is probably my #1 used app.  Also I often got corrupted links in any frameworks that I add to Xcode projects (frameworks often contain links in them).  But only to the apps that I was working on at the time -- none of the symlinks in my other, inactive Xcode projects appeared to corrupt themselves.

As reqeusted:

112    0 0xffffff7f824ac000 0x3000     0x3000     com.bresink.driver.BRESINKx86Monitoring (9.0) <5 4 3>

  144    0 0xffffff7f82519000 0x5000     0x5000     com.trusteer.driver.gakl_driver_2 (1) <29 7 5 4 3 1>

  155    0 0xffffff7f82549000 0x2000     0x2000     com.nvidia.CUDA (1.1.0) <4 1>

Filesystem     512-blocks       Used  Available Capacity   iused     ifree %iused  Mounted on

/dev/disk5    11719722624 2710149632 9009060992    24% 169416350 563066312   23%   /

devfs                 418        418          0   100%       724         0  100%   /dev

/dev/disk4      976101312     875896  975225416     1%    109485 121903177    0%   /Volumes/Extra Disk

map -hosts              0          0          0   100%         0         0  100%   /net

map auto_home           0          0          0   100%         0         0  100%   /home

/dev/disk6s2   7813186512 7707351904  105834608    99% 963418986  13229326   99%   /Volumes/BackupDrive

df-k

new-host-2:/ root# df -k

Filesystem    1024-blocks       Used  Available Capacity   iused     ifree %iused  Mounted on

/dev/disk5     5859861312 1355076120 4504529192    24% 169416513 563066149   23%   /

devfs                 209        209          0   100%       724         0  100%   /dev

/dev/disk4      488050656     437948  487612708     1%    109485 121903177    0%   /Volumes/Extra Disk

map -hosts              0          0          0   100%         0         0  100%   /net

map auto_home           0          0          0   100%         0         0  100%   /home

/dev/disk6s2   3906593256 3853689892   52903364    99% 963422471  13225841   99%   /Volumes/BackupDrive

And in response to dburr - issue appears to only occur to apps that in use.