FC vs. iSCSI performance with Mac OS X Server

Question

FC vs. iSCSI performance with Mac OS X Server

Hi All,

I've relied on direct-attached Fiber Channel storage for our network server volumes since the early days of the Xserve RAID units. We use our network volumes for graphic design needs (with large files) over a Gigabit network, with some Windows clients, too. We're currently running Mac OS X 10.5.8 Server on an Xserve quad core (2008 model) with a 16 TB Enhance Technology UltraStor RS16FS single controller RAID unit with 4Gbit FC. It's worked extremely well for us, but it's going on 3 years old now, and I'm looking to replace it with a dual controller RAID unit for added piece of mind, either from Enhance Technology, Promise, or another vendor. I'll also be increasing the capacity to 32 TB.

The big question is whether I should make the leap to iSCSI with a new RAID unit? I understand the advantages of iSCSI, but I'm concerned about its real-world performance and how taxing it would be on the server itself. Any decrease in performance from our existing 4Gbit FC would be unacceptable. I'm also concerned about the learning curve involved in setting up an iSCSI SAN, since my networking skills are fairly basic -- I really don't get into the advanced functions of our managed switches, and the one time I tried to set up link aggregation, things went horribly wrong! Direct-attached FC sounds a lot simpler.

I'd get a PCIe 4-port Gigabit Ethernet card from Small Tree and use our existing SMC TigerStack II managed Gigabit switch to aggregate the ports, unless anyone thinks a new dedicated switch would be better. I'd likely buy the ATTO iSCSI initiator to make everything happen. I suppose if I wanted to put the old FC RAID unit on the SAN, I could use the

Any opinions, suggestions, or links to benchmarks would be appreciated.

Thanks

Xserve, Mac OS X (10.5.8)

Posted on Mar 21, 2011 11:14 AM

Reply

Answer 1

Mar 21, 2011 12:15 PM in response to Kevin P Poole

If you are pushing your storage (hard), then I'd tend to stay with FC SAN, or look to move to 10 GbE. Or both.

The 2 Gb and 4 Gb SAN connections are faster than GbE, and you already have most of the infrastructure. There's an upgrade to 8 Gb HBAs and FC SAN arrays around for certain configurations, too.

Also look at whether you're going to be using a software initiator, or if you have iSCSI hardware initiator.

If you are looking to add hosts to your storage configuration, you'll likely want to stay inside of what is supported by the Xsan software, too.

Another thing to consider here is how fast you're pushing your LANs secondary to your SANs. Going from a 4 Gb HBA through a host and out a GbE is going to give your bandwidth a haircut. If you hit the wall, you may then need 10 GbE and/or increased network segmenting.

There are vendors which have an upgrade path from iSCSI to FC SAN storage for certain of their storage controllers, but I don't know off-hand if they're (also) supported with Mac OS X. Some of the HP StorageWorks MSA-class gear offers this path, for instance. Others may well have similar offerings.

Reply

Answer 2

Mar 21, 2011 3:36 PM in response to MrHoffman

Thanks for the reply, MrHoffman -- I appreciate your insights. Just to be clear, I'm currently using a 4Gbit FC RAID system directly connected to an Xserve with an Apple FC card, so there's no SAN. I'm sharing out the volumes on that RAID unit via AFP and SMB over the GbE network, so that's really where the bottleneck exists (which is fine, since working off a network RAID volume is still faster for clients than working off their local hard drives).

Ultimately, I would like to create a small SAN, if only to allow my backup server direct access to the main network volumes for D2D backups, and potentially to allow certain client computers direct access to the volumes. It would be far easier to do with iSCSI than with FC, since as you said, I could use my existing network infrastructure.

But in the short term, I'm in a bit of a time crunch, so staying with direct-attached FC certainly would be the simplest option.

Of course, the driving factor here is cost. I definitely can't afford to build a FC SAN, nor can I really afford to hire a consultant to help me establish the optimal iSCSI network environment. Argh!

I'm leaning towards maybe just adding another 16-drive JBOD unit to the existing FC system to meet our short-term storage needs. That doesn't address my desire to get a dual controller unit. But really, how often do RAID controllers fail? I run nightly clones of all my volumes to eSATA arrays, btw.

Reply

Answer 3

Mar 21, 2011 4:47 PM in response to Kevin P Poole

Thanks for the reply, MrHoffman -- I appreciate your insights. Just to be clear, I'm currently using a 4Gbit FC RAID system directly connected to an Xserve with an Apple FC card, so there's no SAN.

Just to be clear, there's a SAN here. You just don't have a SAN switch, based on your description.

I'm sharing out the volumes on that RAID unit via AFP and SMB over the GbE network, so that's really where the bottleneck exists

That's typical, and why I pointed to the GbE as the bottleneck. (If you listen very carefully to the server, you can hear the little skidding sounds as each of the SAN packets decelerates onto the GbE.)

Ultimately, I would like to create a small SAN, if only to allow my backup server direct access to the main network volumes for D2D backups, and potentially to allow certain client computers direct access to the volumes. It would be far easier to do with iSCSI than with FC, since as you said, I could use my existing network infrastructure.

That's a PCI-X or PCIe-class Mac, the Xsan software, a switch, and another connection into your existing (yes, you have one) SAN. As for the array, the prices on those range from Not Too Exorbitant to Oh My Aching Wallet. Used gear (where you can find it) can be a decent investment when you're on a budget.

Xsan has a fixed price, and the SAN switches tend to show up on the used market; they're common in the enterprise space, and a 4 Gb SAN switch is not even remotely new gear.

If you need Big Cheap Storage, then Direct Attached Storage (DAS) approach will be your cheapest option. (You almost have a DAS configuration now.) A PCI-X or PCIe controller or a RAID controller connected out to a Big Dumb Disk Array, err, a JBOD, or into a Big Not-So-Dumb RAID Array. If you have a controller or an open slot in your Xserve box.

I'm leaning towards maybe just adding another 16-drive JBOD unit to the existing FC system to meet our short-term storage needs. That doesn't address my desire to get a dual controller unit. But really, how often do RAID controllers fail?

Um, how often? Usually only when you have no current backups and a deadline, in my experience.

🙂

Reply

Answer 4

Mar 21, 2011 5:20 PM in response to MrHoffman

I was always under the impression that a SAN involved connecting multiple machines to one or more storage device with SAN software, so that the device acted as a local volume. But heck, if my setup fits the definition of a SAN, I'm cool with that! 🙂

Good tip about the used 4Gb fiber switches. A quick search on eBay comes up with quite a few. I'm a little scared about the whole "port licensing" issue when buying a used switch, though, so I'll proceed carefully.

Funny you should mention DAS option: the other PCIe card in my Xserve is filled with a 4-port Highpoint eSATA port multiplier card, attached to two separate 8-bay eSATA enclosures set up in RAID 5 configurations. Those are the volumes that I run clone-style backups to every night. That's been my emergency failover in case the primary RAID system fails, but I still feel like losing a day's work is a risk, which is why I've been wanting to upgrade to a dual controller RAID unit.

I also have separate incremental backups that run on a Mac Pro with Retrospect 8 that does backups to a faux virtual tape library, using a 16-disk JBOD rackmount eSATA enclosure with port multiplication and 2 TB removable HDs. Those backups, especially when I reset a "tape set," would run a lot faster if I had a 4Gbit+ connection to the primary server/RAID volume, rather than the 1 Gb. I did try to aggregate the Gigabit ports on both machines and on my switch to at least get a 2 Gb connection, but doing so took down the switch, so I've been a little concerned about trying it again. D'oh!

Lots to think about, but I guess I'm on the right track.

Reply

Answer 5

Apr 2, 2011 9:14 PM in response to Kevin P Poole

Kevin-

I'm a bit tardy to this thread, but you and I are in the same boat. We are a keep-it-simple Direct-attached AFP/SMB shop, GbE trunk, SMC switch. LOL.

SANs make a great deal of sense for Video Editing, but less practical for traditional file sharing. The cost and increased complexity of a FC switch, metadata controller, and client HBA's make the decision to remain direct-attached a no brainer for me.

I am considering Active Storage (ActiveRAID) as a possible storage solution. FWIW, they have recently announced a metadata controller of their own (ActiveSan). The company seems to have a solid product and support rep in the Mac based SAN market, config the raids to your needs. and I've been impressed with their sales people so far.

Promise vTrak is my other option.

My plan is to keep it simple by staying direct attached with active/active redundant raid controllers, upgrade our PowerMac G5 to a Mac Pro, and aggregate 4 or 6 GbE (Small Tree NIC) to the switch. I will probably do some testing with our current config by aggregating 2 GbE, before committing to that plan. Otherwise I will have to invest another chunk for a new switch with an 10gE uplink. May be worth it if it prevents headache.

Good luck, and let us know what you choose to do...

Reply

Answer 6

Apr 2, 2011 11:35 PM in response to rroselavy

Good to hear what you're considering, rroselavy. I've also been looking at ActiveRAID -- I actually chatted online with one of their reps recently. Since they're so Mac-specific, I asked whether they were considering building a RAID system with Thunderbolt. He basically said that if Apple started shipping all Mac Pros and other machines with Thunderbolt, they'd might consider offering that interface option. It would save a PCIe card!

I think I'll test out gigabit link aggregation again on my own, too. Perhaps over a long weekend in case I screw things up. 🙂 In the meantime, I've decided to get a JBOD unit for our existing single controller FC system, since I can always use it down the road for secondary storage.

Good luck!

Reply

Answer 7

Apr 3, 2011 2:25 AM in response to rroselavy

Using a Apple sold VTRAk Promise dual controller RAID in active-active mode we observed you lose "half" the write performance due to cache updates across the controllers.

Wether that extra security is worth it is your call but in a video editing or other demanding performance scenario it might not be usable and you might have to go to active-passive.

Also with a DAS connected, Promise or other FC RAID, it's just a FibreChannel connected "disk", I can't understand where a SAN comes in.

And one advantage with a (X)SAN is you can grow a volume/filesystem.

Hopefully something like http://tenscomplement.com/ will bring a alternative to HFS+.

Reply

Answer 8

CLWolf

Level 1

25 points

Jul 3, 2012 10:09 AM in response to Leif Carlsson

You are correct. Having a FC Raid attached directly to your FC card is just a DAS. It can easily be made into a SAN, but they way the OP has the initial setup is a DAS.

Reply