Internal RAID 6, six internal drives

Question

Level 1

8 points

Internal RAID 6, six internal drives

I'm doing some research before buying and have been having a heck of a time figuring out just what is and isn't possible (and reasonable) with respect to stuffing drives into a Nehalem Pro. It doesn't help that even though Maxupgrades seems to have some products that would help, they do a lousy job of describing them.

What I want to do:

• Mac Pro with four-drive internal RAID 6 array (hardware RAID with a PCIe card of course), and
• Two-drive internal RAID 1 array (software RAID built with Disk Utility)

Internal because the space I want to put it in won't really accommodate an outboard disk enclosure (although I do want to get a RAID card with capacity for external drives as well in case things change).

Anyway, there are many variables here to be figured out (what RAID card for example, Highpoint and Areca seem to be the most likely contenders) but I'm having a surprisingly hard time figuring out just what's possible with respect to putting six drives in a Nehalem Pro. As far as I can tell the straightforward option is:

• Maxupgrades "backplane bypass" sleds, http://www.maxupgrades.com/istore/index.cfm?fuseaction=product.display&product_i d=189, four of them. I'm guessing these just stand off the drives from the backplane (or turn them around?) and expose their connectors so they can be hooked to good old fashioned SATA cables. But, of course Maxupgrades doesn't say.
• Maxupgrades two-drive optical bay mounting kit, http://www.maxupgrades.com/istore/index.cfm?fuseaction=Product.display&product_i d=187. Comes with a PCIe SATA card because the Nehalem Pro doesn't have enough SATA ports.

However, it occurs to me that instead of throwing away the built-in SATA of the four bays (because of using the bays for my RAID 6 drives) and then putting a PCIe SATA card in to add back in a couple SATA ports for the two RAID 1 drives, why couldn't I do this instead:

• Only two "backplane bypass sleds".
• Maxupgrades optical bay mounting kit without PCIe SATA card, http://www.maxupgrades.com/istore/index.cfm?fuseaction=product.display&product_i d=158. Cheaper of course.

The idea would be to make the two drives in the optical bay, plus the two drives in the "backplane bypass sleds", into the RAID 6. I'm not positive this makes sense (a) at all, and (b) from a cable routing point of view. Does it? (Yes the drives get laid out kind of funny in the chassis, but not putting an extra PCIe card into the box seems like an obvious win.) Then the two RAID 1 drives would just plug into the backplane like Apple intended. Occupies fewer slots, uses less power, and cheaper too.

Of course, yet another option would be if I get a used 2008 Mac Pro, that has a MiniSAS connector on the logic board I can plug my RAID 6 controller directly in to. But then I wouldn't have the LATEST AND GREATEST, now would I? :-/

Posted on Jun 26, 2009 5:58 PM

Reply

Answer 1

mbean

Level 4

1,250 points

Jun 26, 2009 11:40 PM in response to jgs42

The idea would be to make the two drives in the optical bay, plus the two drives in the "backplane bypass sleds", into the RAID 6. I'm not positive this makes sense (a) at all, and (b) from a cable routing point of view. Does it? (Yes the drives get laid out kind of funny in the chassis, but not putting an extra PCIe card into the box seems like an obvious win.) Then the two RAID 1 drives would just plug into the backplane like Apple intended. Occupies fewer slots, uses less power, and cheaper too.

Hi,
The problem that I found with an internal RAID 6 configuration was .... RAID 6 uses the space of two hard disks for parity data. That means that a six drive system is really only a four disk RAID which reduces performance. In addition, an internal six disk RAID6 will make the computer run warmer and louder than expected and there is no way to run Boot Camp (without reflashing the controller card BIOS each time you want to boot from Mac OS X to Windows) as none of the RAID 6 cards that I have installed will support Boot Camp with the bootable Mac OS X EFI installed. You need one of the internal HDs for that. Add to that the fact that every once in awhile an update comes out that cannot be applied with a RAID boot disk and you start to see why I really like having a single boot drive available when needed.

The solution for me was to setup an 8 drive external bootable RAID 6 outside of the Mac Pro. This allows me to use the RAID 6 when needed. I only need it for certain high bandwidth tasks - not all the time. This saves energy, provides higher performance, keeps the Mac Pro cooler, reduces wear on the disks (as they are only on as needed), allows me to use an internal as a spare boot disk and for boot camp and I can easily see where a hard disk failure occurs by looking at the HD tray lights. Plus with hot swap drive trays I can set up different RAID sets for different projects. In other words, I get all the power of a large RAID 6, with much more flexiblity and I do not have to run it when it's not needed.

This is a nice 8-bay combination:
The HighPoint RR4322 and the Enhance E8ML
http://www.amug.org/amug-web/html/amug/reviews/articles/highpoint/4322/
http://www.amug.org/amug-web/html/amug/reviews/articles/enhance/e8/
Eight Samsung HDs in RAID 6 = 488MB/sec. read and write on average.
(Averaged from 0% to 100% full).

If you really want a HUGE high powered RAID6 its hard to beat the HighPoint RR4322 and the Enhance UltraStor RS16 JS SAS/SATA 16 Bay RAID Storage System.
http://www.amug.org/amug-web/html/amug/reviews/articles/highpoint/4322/
http://www.amug.org/amug-web/html/amug/reviews/articles/enhance/rs16js/
Sixteen Samsung HDs in RAID 6 = 696MB/sec. write and 833MB/sec. read on average.
(Averaged from 0% to 100% full).

You can try an internal RAID 6 setup, but I did not want to loose the flexibility of using Boot Camp, HD access lights (for failure monitoring) and the ability to turn off the RAID 6 when I only needed a single boot disk for more normal computing. 🙂

Happy hunting!

Reply

Answer 2

jgs42 Author

Level 1

8 points

Jun 27, 2009 2:59 AM in response to mbean

Thanks for the reply. I actually was considering dividing the six internal disks into a FOUR disk internal RAID 6, and a two disk internal RAID 1. The RAID 6 would be just for data. The RAID 1 would be the boot volume and would use the native Apple software RAID. This ought to address your good point regarding not wanting to boot off the RAID 6. As for Boot Camp, I really don't care about it -- had it installed on my last machine and after I didn't use it in over a year, deleted it to recover the space.

I do realize that a four-disk RAID 6 sacrifices fully 50% of the space to parity. My requirements (or desires, really) are perhaps somewhat unusual in that redundancy/reliability are more important to me than performance. I do want decent performance, but for the time being I would find 125 Mbytes/sec to be adequate since much of the access to the volume will be over a network anyway. My feeling was that if the four disk RAID 6 gets to be either too small, or too slow, I can add disks to the set at that time (probably would have to do it externally since IIRC the cards I'm looking at just support four disks internally).

Thanks for the reccs on cards and enclosures.

Reply

Answer 3

noice_T

Level 3

545 points

Jun 27, 2009 4:05 AM in response to jgs42

Is this MP running as a server? I only see raid 1 for boot making sense if the comp is running 24/7 as a server. If its for an everyday workstation, perhaps using the 6th drive as a backup drive would serve you better.

Also just from my personal experience, I hope you have +at least+ 1 daily backup of the raid 6. My Sonnet D800RAID running a raid 5, had 2 drives die 10 mins apart from each other...it took the raid to pieces. I know you're running a raid 6, so you'd still be safe, but it's still scary. There is no substitute for backups.

Reply

Answer 4

The hatter

Level 9

61,031 points

Jun 27, 2009 4:45 AM in response to noice_T

I would suggest then that you look at using SoftRAID.com mirror RAID for 4 drives. You get 'stripe read' speed and all the redundancy. Might even settle for 3 drives.

I don't think it makes sense otherwise.

Some people do use a mirror (2 drives) when doing backups. Actually, 35 yrs ago we would dump disk to tape and output two tapes concurrently - just didn't know it was 'mirror' and was because you could count on tape to fail, and we couldn't afford that.

Reply

Answer 5

noice_T

Level 3

545 points

Jun 27, 2009 5:21 AM in response to The hatter

Stripe read...very cool~ nice tip once again.

Lol, tape. I wish tape wasn't that expensive and was just faster in general. We have two old exabyte vxa-2 tape drives at work, and a lot of old tapes. I was thinking of using them for offsite backups to a safety deposit box, but man...it just crawls. Maybe it's just the drawback of old technology, but I nearly took an entire day to backup like 500GB. I guess the types of files we have just don't compress well..

Reply

Answer 6

jgs42 Author

Level 1

8 points

Jun 27, 2009 10:43 AM in response to The hatter

I can't see mirror (i.e. RAID 1) as being the equivalent of RAID 6 in terms of redundancy. RAID 1 is 1+1, so a double fault will fail the set. RAID 6 uses double parity, so the set can survive the failure of any two disks. Also, how do you figure 3 drives in a RAID 1 set? Seems to me you need an even number of disks for RAID 1. Or are you suggesting two drives in a 1+1 configuration and one non-redundant disk? It is true that the usable storage you get from 4 disks in a RAID 6 set is the same as you get from the same 4 disks in RAID 1, but you get better redundancy and if you grow the set to more than 4 disks, you get more storage capacity out of RAID 6. On the down side, you do have to pay for a pricey hardware RAID controller.

Oh BTW if Wikipedia is to be believed :-/ the read performance on a RAID 6 set of four disks should be almost the same as a RAID 5 set of three disks or a RAID 0 (stripe) set of two disks. In general, read speed on RAID 6 N disks ~= RAID 5 (N-1) disks ~= RAID 0 (N-2) disks. Not quite, but almost.

I remember tape. I have a pile of Exabyte and DAT tapes in the corner of my office and nothing to read them with any more. I even have some old Teac streaming cassettes!

Reply

Answer 7

jgs42 Author

Level 1

8 points

Jun 27, 2009 10:51 AM in response to noice_T

It is a server, although only for a small group so I don't really need 24/7 uptime. The RAID 1 is more out of laziness than anything else, to be honest -- I really don't want to spend the time or effort to do a reinstall-and-restore if the boot volume croaks. I'm willing to spend the cost of an extra disk to avoid having to do that.

And I couldn't agree more about backup. I would add, do the backup onto a separate machine if possible instead of a locally-attached disk. Ideally, a geographically diverse machine (I haven't achieved this yet). In addition to the possibility of multiple faults as you describe, I've also been saved by my backup several times when an app (or the boneheaded user, namely me) has damaged or deleted data. A RAID set will do just what it's told -- if the app writes bad data to the RAID set, game over. Same if you delete a file you shouldn't have. That's when you dig into the backup. I would also add, a backup that keeps some history, like with Time Machine or Retrospect. Some folks swear by simple duplicates made with SuperDuper or CCC, but then what do you do if you discover one day that a file got corrupted a month ago? This has happened to me.

Backup backup backup. And then hope you never have to use it. 🙂

Reply

Answer 8

jgs42 Author

Level 1

8 points

Jun 27, 2009 10:56 AM in response to jgs42

By the way, I do appreciate all the replies so far, but what I'm hoping for the most is someone with experience with the Nehalem Pro and Maxupgrades stuff to comment on whether my scheme works. Just to sum it up, that scheme is:

• Two drives mounted in "backplane bypass sleds"
• Two drives mounted in optical bay
• PCIe RAID controller connected to the four above drives

plus

• Two drives in backplane bays as shipped by Apple

All feedback is appreciated but it would be really cool if someone knows the answer to the above. Anyone, anyone at all, Bueller?

Reply

Answer 9

The hatter

Level 9

61,031 points

Jun 27, 2009 11:22 AM in response to jgs42

And which RAID card to you plan to use? that wasn't mentioned (that and the details of how you planned to set it up).

I'm still opposed to 3rd party card. OS X updates tend to be problematic. Snow Leopard support. And no fan of the Apple RAID card.

Reply

Answer 10

jgs42 Author

Level 1

8 points

Jun 27, 2009 11:57 AM in response to The hatter

As for which RAID card, undecided as yet. Several of the Highpoint RocketRaid 35xx series would appear to do the trick. Yes I know only the 3522 is listed as supported on the Mac, but as this Barefeats article ( http://www.barefeats.com/hard120.html) points out, it's only because Highpoint assumed there was no way to get access to the internal drives. The Areca 1212 and 1222 both look possible (though no SFF-8088 is a bummer). Most of the Areca 16xx series look like they'd do it too. This is just with respect to supporting Mac OS X, supporting RAID 6, and having at least one SFF-8087. I haven't looked (much) at pricing or performance yet, though as I mentioned earlier my needs for performance are probably less than many others'.

I do have some past experience with the dangers of OS upgrades and third-party RAID cards. I plan to be a little more cautious this time around, in particular by using a plain vanilla Apple disk configuration as my boot volume, and by not trying to be the first kid on the block to do an OS upgrade. (And the Apple RAID card is not on the list -- no RAID 6 support.)

Reply

Answer 11

mbean

Level 4

1,250 points

Jun 27, 2009 12:36 PM in response to jgs42

I can't see mirror (i.e. RAID 1) as being the equivalent of RAID 6 in terms of redundancy. RAID 1 is 1+1, so a double fault will fail the set. RAID 6 uses double parity, so the set can survive the failure of any two disks.

Hi,
The downside to any RAID including RAID 6, is should the directory become corrupted you can still lose the volume. This is probably one of the biggest issues with a RAID 1 mirror. No matter what RAID set is created you still need a backup (preferably off line) of any important data. Too many users do not understand this concept.

It is better not to learn this from experience 🙂

Reply

Answer 12

The hatter

Level 9

61,031 points

Jun 27, 2009 12:57 PM in response to mbean

I didn't suggest using just two drives for a mirror. It isn't "1+1".

I have yet to hear of anyone, ever, using 3 drives in a mirror. Which to me is senseless almost unless fora backup set, where I think two can be a good idea - for ONE of your backup sets, and not the sole set.

Also, SoftRAID maintains a backup directory and volume table, just in case.

Reply

Answer 13

jgs42 Author

Level 1

8 points

Jun 27, 2009 6:11 PM in response to The hatter

Regarding "1+1" I was using it in the sense of levels of redundancy in a fault-tolerant system (http://en.wikipedia.org/wiki/N%2B1_Redundancy, "1+1 Redundancy, which is a system configuration whereby each critical component has a redundant backup component to ensure component-level continuity.") Sorry for using the wrong buzzwords! I know RAID jargon tends to use "N+M" terms in a way I find unfamiliar. My point was just that in a RAID 1 set you burn (at least) one mirror disk for every disk of usable filesystem capacity, and a double disk failure can fail the whole set (if you have bad luck and you lose a disk and its mirror). With RAID 5 you burn one disk, period, and with RAID 6 you burn two and any two disks can fail without killing your filesystem. (But there are other disadvantages of course.)

Reply