Ambiguous RAID failure

Sorry, which messages exactly are you asking about?

The messages from the log in RAID Utiliity pertaining to this (got them by Exporting the log) were, in reverse time order (the 8:22 entry coincided with this morning's restart):

Friday, May 18, 2012 08:22:38 ET Degraded RAID set RS1 - No spare available for rebuild critical

Friday, May 18, 2012 03:51:09 ET Degraded RAID set RS1 - No spare available for rebuild critical

Friday, May 18, 2012 03:49:53 ET Degraded RAID set RS1 warning

Friday, May 18, 2012 03:49:51 ET Drive 3:5000cca216e13980 failure detected - Previous drive status was inuse critical

I can't find anything in the logs pertaining to the status of all the other drives, except that there are a few hundred "AppleRAIDCard" entries in kernel.log starting at 3:19, reporting scsi_request and scsi_task errors. Those stopped at 3:51 and have not recurred. I can post some (or all) of those if it would be helpful.

Thanks.

Yeah, I had noticed that it said "Drive 3:xxxx" and realized that there was an ambiguity there. I might quibble with the contention that it's a "huge" assumption to think it's Drive 3 since in the past I've had such messages in which the number in the message did correspond to the bay number of the failed drive. But that could have been coincidental.

I'm going to take your advice and replace the drive in Bay 1 as soon as the bootable backup that I'm making via SuperDuper completes. (I have a TM backup but would prefer to have an easily bootable image in case something else goes wrong.) I'll report back.

One other question: the Hitachi drive in Bay 1 (the presumably failed drive) was supplied by Apple with the Mac Pro four years ago. Is there any reason to seek a replacement from Apple rather than from Hitachi? (Assuming the drive is still under warranty at all. If it were a Seagate, it might still be under warranty.)

Thanks for your help.

Update: even with the failed drive removed from the RAID Set the system was too unstable and wouldn't stay up long enough to complete a full backup with SuperDuper.

So I replaced the drive in Bay 1 anyway and all is well so far. Marked it as a spare and the rebuild has begun, If past experience is any indication it will take about 72 hours to complete.

The RAID rebuild is happening much more quickly than it has in the past. If the current speed is any indication then it will be done in closer to 24 hours than the usual 72 hours.

This is the first time that the system has had four matched Seagate drives in it; before now there were always three Seagates plus the Apple-supplied Hitachi that came with the system.

This prompts me to ask: does this suggest that the previous configuration was sub-optimal because of the mismatch? Does it suggest that the Hitachi might have been problematic all along?

RAID array was rebuilt in under 24 hours and RAID Utility reports that all is fine.

Unfortunately my system is still misbehaving in the same way: unpredictable and frequent freezes, spinning beachball, etc. It did this several times during the rebuild (requiring a reboot) and a couple times since.

In the past when I had these symptoms, replacing the ultimately-failed disk drive solved it. So I'm not sure how to troubleshoot this further. I don't see anything obvious in the logs under /var/log that would point to the problem (like a ream of messages about a problematic disk drive) but perhaps I'm not looking closely enough.

Thanks, but it doesn't seem to be related to swapping/paging or to CPU usage by a runaway process. The system always seems to have plenty of RAM (18 GB is installed) and neither Activity Monitor nor iStat Menus ever seem to show anything obviously suspicious (nor does "top" when I run that), No swapping, no processes consuming all the CPU (also, it's an 8-core machine).

What typically happens is that the system will seem fine, but then running a new command, or (e.g.) asking my IMAP client to move some files, provokes a hang: everything locks up, spinning beachball, etc. Sometimes this is preceded by an obvious degradation in performance, but sometimes not.

It "feels" disk-related and in the past I had similar symptoms preceding a disk failure (as I did before replacing the drive a few days ago). But I can't see anything relevant in the log files, and in the past, when a disk failed, I didn't see clear evidence of this in the logs until RAID Utility declared the failure--nothing previous to that.

Is there some way to increase the verbosity of the disk-related logging? If a disk really is problematic, even if the problems are recoverable after retries, you'd think that would be detectable by the system and that it could be logged.

The only other things I can think of trying right now are either: (1) replacing the remaining drives in the RAID Array one by one, starting with the drive in Bay 3 (because of the earlier ambiguity), and/or (2) disconnecting as much as possible from the system and seeing if it becomes more stable in that configuration. Though I've already mostly done (2) without any obvious improvement,

Thanks. Scanning for bad blocks does seem like a good idea. I don't have Tech Tool Pro, Drive Genius, or Disk Warrior; they're each about $100 but I'll happily buy whichever one of them is likely to be most useful here. Would that be DG? Mr. Hatter doubts that TTP will be as useful here, and Drive Warrior sounds like a higher-level recovery utility, operating at the file system level rather than at the level of blocks.

But it also sounds like I could try Seagate's SeaTools first to achieve the same thing, is that right?

And can all these utilties scan for bad blocks without also reformatting the drive at the same time? And on a live system?

I've considered replacing all the drives with newer ones but of course it's pricey to do that.

Let me ask this: my Netgear ReadyNAS Pro allows drives in the array to be replaced with higher-capacity drives, and the array will be expanded to take advantage of the additional space. I guess it's safe to assume that the Apple RAID Card has no such capability?

In which case, if I want to grow the array, do I need to replace all the drives with higher-capacity drives (2 TB or 3 TB), recreate the array using RAID Utility, and then restore my system from backup?

I've chosen Seagate drives in the past because their warranty replacement process is so straightforward; essentially if a drive is in warranty they'll just replace it without making you jump through hoops. Not sure if other manufacturers are as easygoing in that respect.