4Gb Xserve MDCs Fibre Disconnect Randomly

Question

4Gb Xserve MDCs Fibre Disconnect Randomly

We recently upgraded to Xsan 2.2 with two new Intel Xserve MDCs with 4Gb fibre cards. Now every few days, the MDCs seem to lose their link to the fibre switch. The switch itself shows that they're connected and the link lights on the fibre card are showing normal status, but in the Fibre Channel preference pane, it says "No Link Established". I have to unplug and replug the fibre cable to get it to come back. This doesn't happen on any of the client machines.

Here's my configuration:

QLogic SANBox 5200 Fiber Switch
Two Intel Xserves with 4Gb Fibre Cards, using copper cables to the switch
Xserve RAID (Late 2004) with 2Gb Fibre cards, using copper cables to the switch
Several clients with 2Gb Fibre cards, using fiber optic cables to the switch

I found this thread that says their may be compatibility issues between the 4Gb cards and the old 2Gb Xserve RAIDs, but I can't find any solutions. http://discussions.apple.com/thread.jspa?threadID=2319309&tstart=-1

Thanks!
Jason

Mac Pro, Macbook Pro, et al, Mac OS X (10.6.3)

Posted on Jun 16, 2010 5:33 AM

Reply

Answer 1

jmyres

Level 1

80 points

Jun 22, 2010 10:30 PM in response to Jason Buckner

It sounds like you need to reboot the machine that's showing you no link in the fibre preference pane. Whenever we unplug, change cables, or adjust settings for a given fibre port or connection, we also reboot that machine to make sure the changes have taken place.

If you log in to your Q-logic switch do you see any CRC or Decode errors on the ports might be giving you a problem? If you see more than a few decode errors, or more than about 100 CRC errors, you may have a problem with a cable or SFP. Try resetting the counter, and then watch to see if the errors build up again over a period of time. After a counter reset, if there is a problem with one of your connections it will standout fairly quickly.

If you are using fibre make sure you are not exceeding the bend radius (4x the thickness) of the cable or pinching it anywhere. If you are using copper, make sure it is not being run in parallel with power cables or other noise sources.

We have never had problems using 4Gb gear with Xserve RAIDs.

JM

Reply

Answer 2

Jun 24, 2010 7:59 AM in response to jmyres

We just had another incident of the fiber being lost and when I restarted the MDCs, I was still getting No Link Established. It wasn't until I restarted the XRaid that I was able to get the MDCs to get their Link Established.

I noticed that one of our MDCs does have an abnormally high Decode Error count (355, though I don't know the time period for this) so we're going to try swapping cables on that one to see if we found our culprit.

Thanks for the tip on the error count. I'm going to keep an eye on that.

Jason

Reply

Answer 3

Jul 15, 2010 7:58 AM in response to Jason Buckner

I am having exactly the same issue!

We have an old Apple Xraid, with two new Intel Xserve working as primary and secondary metadata controllers. All using Xsans 2.2.
We have six brand new Mac Pro's in our environment, which are all showing the "No link Established" message under system preferences.

Any solutions would be great! or any word from apple saying there is in fact a compatibility issue would be greatly appreciated

Reply

Answer 4

Jul 15, 2010 8:36 AM in response to David Kahn3

Just an update, I have sent Apple Enterprise Support a request for help regarding this issue. Hopefully they can shed some light (no pun intended) on the matter.

Reply

Answer 5

Jul 21, 2010 6:19 AM in response to David Kahn3

Hello all,

Apple suggested this and it worked for me. In short there is an known issue with mixing 2GB and 4GB devices. I set EVERYTHING, servers, and clients to 2GB and Point to Point, these resolved the issue.

From Apple:

1. Hardcode speed and topology on all machines
- in the Fibre channel utility set speed to 2Gb and topology to Point to Point
- reboot

2. verify switch is configured according to:
http://support.apple.com/kb/HT1084

3. Use 4Gb SFP-SFP cables
* We have seen issues with 2Gb cables with 4Gb HBAs

http://store.apple.com/us/product/MA461G/A
* Article to help identify cables if needed: http://support.apple.com/kb/HT1769

Hope this helps

Reply