New RAID array constantly rebuilding

Question

Level 1

51 points

New RAID array constantly rebuilding

I've just bought a Mac Pro in the stock 2.8GHz 8-core configuration and kitted it out with 4 HDDs (Samsung Spinpoint). They're arranged of 2 volumes, each consisting of two identical size drives set up for mirrored RAID using Disk Utility. DU was run directly from the factory-supplied Leopard installer DVD, since I was replacing the factory-supplied system drive and rebuilding the system from clean.

Trouble is, Disk Utility is always listing the two RAID volumes as degraded and claiming that they're rebuilding. I left it overnight at one point and both rebuilds completed, but after a small amount of use of the system that day, sure enough the boot volume went back to rebuilding (the progress bar starting from scratch and the estimated completion time measured in days), followed a while later by the second volume (used just for data). Right now, both are sitting with progress bars at about 25% completion with an estimated 6 hours (boot volume) and 3 hours (data volume) for completion.

Surely, as I write data to either volume, data is simultaneously automatically written to the mirror? There should be no need to rebuild unless a drive has failed and been replaced. Is this normal behaviour? Are there other steps I can take to diagnose the fault? As it stands, the system never seems to catch up and finish rebuilding the arrays, and it can't be doing much for the longevity of the drive pairs, when (provided the machine is idle) one is constantly being asked to read at flat-out rate while the other's writing as fast as it can.

Many thanks for any advice.

MacPro early 2008, Mac OS X (10.5.2)

Posted on Mar 17, 2008 12:19 PM

Reply

Answer 1

islk350

Level 1

65 points

Mar 17, 2008 1:05 PM in response to Pond

Try searching on a knowledge base articles are raid. Not sure of the size of your raid system but Id also check on any limits at 2T or 4T sizes. Maybe they need to be partitioned...

Reply

Answer 2

The hatter

Level 9

61,031 points

Mar 17, 2008 1:10 PM in response to islk350

???

How about checking whether there is a size limit (there isn't and 4 x 1TB is supported, even as stripped RAID of 3.6GB). A mirror is going to be 933GB.

Reply

Answer 3

The hatter

Level 9

61,031 points

Mar 17, 2008 1:19 PM in response to Pond

In order to rule out problems and get to the bottom, and something I do with any new drives before I build a RAID is to test and format each drive separately.

Do you have the original drive now in an external case so you can use it off FW or eSATA or the ODD ports? otherwise, for now, might want to boot off that and format each of two drives, insure they work. Do some benchmarks.

Are these retail or OEM labeled?

Once you are sure you have a working pair, and they are not wildly different in I/O (some drives actually are) because a mirror really depends on every drive in the array not having to wait on the other.

One person tried using a WD 1TB "Green" drive along with 3 Seagates and it would only work if there were a total of 3 drives installed. They ordered an RMA replacement, and I think it was OEM.

I've read good things about the Samsung F1s but one person on forums.macrumors.com did have to return one and another had one that was noisy and "chatter" when used (every few seconds).

When in doubt, I zero the drive. With SoftRAID.com you can begin to zero a drive and stop at any time - safely. Don't know how forgiving Apple DU is of the same. There are other programs (Intech Speedtools) that can bench, test, and map out bad blocks (or run tools in Windows to check drive, update firmware if needed etc).

No - they should not be degraded like this. SoftRAID will alert on any I/O error. You might want to see if there are any read or write errors or excess retrys in the system.log.

At first I thought you must have the Mac Pro RAID card which takes hours to setup, charge battery, etc. That is what it sounds like.

Reply

Answer 4

Pond Author

Level 1

51 points

Mar 17, 2008 2:46 PM in response to The hatter

Thanks for the responses so far.

These are OEM 750GB units of identical part number all purchased from the same retailer in one transaction. The typical price of 750GB drives dropped sharply recently, and it's now at round about the cheapest price per GB for standard consumer drives. All four drives seemed to format OK and there don't seem to be any particular performance problems (e.g. the RAID rebuild proceeds at a rate reported by iStat menus which indicates saturation of the 300Mbps bus). I have not, though, done any kind of soak testing on the individual disks.

FWIW I've no problem with noise. You can hear drive chatter when the hard drives are working if you listen carefully, close to the machine, sat in a quiet room. Right now, the PC's on as well - its fan noise drowns out the Mac by quite some margin. The MP seems to be an amazingly quiet machine, even with all four drive bays and all 8 RAM slots occupied; temperatures seem low, the fans are running at the lowest speed, all good it would seem - except for the RAID rebuilds. Well, that, and I've not even bothered trying to use sleep rather than just shutting down, since I get the impression that it's just going to reboot afterwards 😉

Having posted about this, I was sat around thinking about it and suddenly remembered that when I set up the RAID array I was digging around in the options and noticed an "automatically rebuild array" option. I'm pretty sure I set that - yes, should've mentioned this earlier, but I only just remembered - however, the Help pages imply that rebuilds should only occur if something has gone wrong. Perhaps there is a bug here, or the Help pages are misleading and for some reason the option enables continuous rebuilding of the set - though this would seem to be fairly pointless.

Reply

Answer 5

Pond Author

Level 1

51 points

Mar 17, 2008 2:49 PM in response to Pond

I also should mention that these are SATA drives, not SAS, in case it makes any difference. That's why I'm using the OS X built-in RAID. I was hoping to use RAID-5 over three 750GB drives leaving the boot drive out of the RAID set and backed up via Time Machine, but Leopard's software RAID doesn't offer that option - hence the use of mirroring instead.

Reply

Answer 6

The hatter

Level 9

61,031 points

Mar 17, 2008 3:37 PM in response to Pond

There aren't any SAS drives in that capacity, or w/o high end RAID controller.

My advice, never use OEM drives, they don't always have the same firmware as retail.

Why not use a drive for TimeMachine? most of my drives are RAID0, and even my TM volumes are known to be stripped.

Reply

Answer 7

Pond Author

Level 1

51 points

Mar 17, 2008 3:59 PM in response to The hatter

I'm using RAID rather than TimeMachine because it operates at a lower level and has been around a lot longer, so ought to be more reliable and may be more efficient. It also means no complex recovery procedure per se, since it should just be a case of "replace failed drive and rebuild", with no interruption of available data. Finally, I'll be using the machine a lot for audio work, which means lots of big files kicking around. I don't need Time Machine getting all worked up over backing up version after version of some temporary WAV I've recorded, nor do I want the potential for unpredictable loads on the machine during TM backup sessions, rather than a theoretically constant and predictable overhead of a constant mirror operation for all disc writes.

Time Machine's aimed at solving a different problem in many respects - it's about recording and recovering system state, rather than attempting to preserve overall reliability/uptime.

Reply

Answer 8

The hatter

Level 9

61,031 points

Mar 17, 2008 4:08 PM in response to Pond

RAID is not a backup or substitute. Mirrors are not backups. TimeMachine is just one backup method and strategy. So is cloning with SuperDuper or Carbon Copy Cloner. And most should be using both probably, a working bootable backup, and in case one set fails.

I would never put all my eggs in one basket, whether that be TM or RAID mirror. All my my RAIDs are stripped, I want the speed, and I have at least 3 backups for every volume. Recovery strategies generally means redundant backups.

TM is better now than it was, but I keep at least two TM sets as well. I would not rely on just one of that either.

And Apple use to recommend excluding some files from TM. I think now Aperture and TM do that automatically. Same with Spotlight. YOu never want Spotlight enabled on a scratch volume used for editing, it really hurts CS3 performance and caused problems.

I have no idea where you got the idea of using one instead of the other or not using it. Except I don't think mirrors are always the best use. They are useful for live audio editing.

Reply

Answer 9

Pond Author

Level 1

51 points

Apr 2, 2008 12:45 PM in response to Pond

For the sake of search engine archives - this looks like it was down to the Samsung Spinpoint 750GB drives being, politely, junk. See http://discussions.apple.com/thread.jspa?threadID=1314886 for a discussion about the 750GB and 1TB parts.

NewEgg has numerous reviews describing the sorts of problems I ultimately saw myself - the drives would not be recognised at restart or, as the problems worsened, would disappear off the SATA bus at run-time. Since some of the NewEgg reviewers mention that the drives also suffer from in-transit data corruption, the RAID array rebuilds may well have been down to unreported errors when getting data to or from the drives. SoftRAID lists error reporting among its many features and may be a good option for peace of mind if the bare bones Apple RAID doesn't give enough information; I've not yet tried it myself.

Having managed to get all four Samsung drives in sync and online, I replaced them with 750GB Western Digital RE2s, two at a time (starting with the least reliable drives), rebuilding the mirror from the other two at the command line in the OS X installer environment. 10.5.2's Disk Utility has a bug that prevents it rebuilding the mirror to clean drives, doh; see http://discussions.apple.com/thread.jspa?threadID=1212964 and Robin Daugherty's first message in particular. Afterwards the machine wouldn't boot from the rebuilt mirror but a permissions repair seems to have solved it (see http://discussions.apple.com/thread.jspa?threadID=1468605 for more information).

Hopefully that'll be an end to the problems. The machine has only been running from the new drives for an hour or two, but from here on, no news is good news.

Reply