ATTO R380 EXPRESSSAS HBA - unavailable disks in RAID via ATTO CONFIGURATION

Hi All that use the *ATTO R380 EXPRESSSAS PCIE HBA (host buss adapter)* in a 2007 MACPRO dual quad core (8core) machine (1st REV) that have exerienced issues with their logical volumes being OFFLINEd by the ATTO R380 disk array controller and disk array DDMs (disk drive modules) being made unavailable status by the ATTO HBA. (+formatted 1st para for newsfeeders+).

I am posting this as a solution (albeit proper usage) | work instruction and configuration of the ATTOTECH R380 EXPRESSAS HBA including the diags that I encountered so that these can be searched in this forum for help to others.

After recently adding some new WD1002FBYS 1TB DDMs 512 byte sectors (disk drive modules) to replace 4 in to my PROAVIO EB8MS JBOD disk enclosure I experienced numerous errors from the OSX kernal and then further by the ATTO EXPRESS SAS HBA configuration alert software, rendering the logical volume(s) offline an inaccessible.

I experienced these errors when performing large copies of objects (say 2TB-3TB ata time) from or too the ATTOTECH R380+PROAVIO enclosure configured as:
• 2 x 4TB RAID0 logical volumes (1MB interleaf and 4K sectors, prefetch=3) as 8 x sas targets
• these two RAID0 volumes *configured as HFS+ (extended) JOURNALLED*.

Copy operations at 2TB-3TB a time of these large objects (directories of DVCPROHD and PRORESSHQ objects) were:
• conducted using the OSX FINDER (drag and drop) and the shell *unix cp -Rv* command.
• to and from a third (3rd) software raid0 inside the MACPRO (3 x 1TB WD1002FBYS DDMs using apple software raid)
• to and from (between) the two RAID0 logical volumes configured and maintained by te ATTO R380 expresssas HBA+PROAVIO EB8MS jbod enclosure

FAILURE SYMPTOMS EXHIBITED were:
• ATTO EXPRESSSAS event RAIDGROUP <raidgroup_name> status changed form ONLINE to OFFLINE (this is flashed up on the OSX screen
• /var/logs (console.app): shows errors with JOURNALLING failures as

+kernel[0]: jnl: disk6s2: flushing fs disk buffer returned 0x6+
+kernel[0]: disk6s2: media is not present.+
+kernel[0]: jnl: disk6s2: do jnlio: strategy err 0x6+
+kernel[0]: jnl: disk6s2: write journalheader: error writing the journal header!+
+kernel[0]: disk6s2: media is not present.+
+kernel[0]: jnl: disk6s2: do jnlio: strategy err 0x6+
+kernel[0]: jnl: disk6s2: end_transaction: only wrote 0 of 200704 bytes to the journal!+
+kernel[0]: disk6s2: media is not present.+
+kernel[0]: disk6s2: media is not present.+
+kernel[0]: jnl: disk6s2: close: journal 0xc5fac18, is invalid. aborting outstanding transaction+s

• interesting and regular and copius whinging about the FAN control from the ATTO fan controller as:
+fan-controllerd[58]: Setting fan 1 minimum RPM to 1500.+
+fan-controllerd[58]: Failed executing command, key = "F1Mn", error = e00002c1.+
+fan-controllerd[58]: Setting fan 1 minimum RPM to 1500.+
+fan-controllerd[58]: Failed executing command, key = "F1Mn", error = e00002c1.+
+fan-controllerd[58]: Setting fan 1 minimum RPM to 1500.+
+fan-controllerd[58]: Failed executing command, key = "F1Mn", error = e00002c1.+

THe above message always occurred some time after the copy operations started. So one may assume the ATTO EXPRESS SAS HBA card is getting a tad warm doing all this work.

• offline messages from ATTO EXPRESSSAS alerts caused the finder and shell cp cmd to stop in their tracks and then the usual "device pulled offline error message" from the OSX file system.

The message in the ATTO CONFIGURATION TOOL/PCI MEMBER/RAID shows that of the failed RAIDgroup that was now offline with a RED exclamation mark against it whoes usuall one of more UNAVAILBLE status DDms. These could NOT be used again for a RAID group!
Immediatly one feels these drives are faulty which is NOT the case. Additional SMART DIAGS turned on revealed no issues.

USing the ATTO CONFIG TOOL DumpEventLog there actua SCSI WRITE x'002a' failures on several (in the following example 3 targets failed) of the DDM SAS targets (refer 5001086000000fn) where labelled as:

dumpEventLog
+ATTO ESAS Event Log+
+RAID Group state now Online: raidzero01.+
+RAID Group state now Online: raidzero00.+
+1 Retries of Cmd on 5001086000000f27 0, Op: 0x2A+
+1 Retries of Cmd on 5001086000000f26 0, Op: 0x2A+
+1 Retries of Cmd on 5001086000000f24 0, Op: 0x2A+
+Command Timeout on Disk [5001086000000f24 0]+
+Resetting Disk [5001086000000f24 0]+
+Error on 5001086000000f24 0, Op: 0x2A, LBA: 0x370fb000+
+SenseKey: 0x0B, ASC: 0x00, ASCQ: 0x00+
+Retry the operation+
+RAID Group state now Offline: raidzero00.+
+Command Timeout on Disk [5001086000000f26 0]+
+Error on 5001086000000f26 0, Op: 0x2A, LBA: 0x370fb000+
+SenseKey: 0x0B, ASC: 0x00, ASCQ: 0x00+
+Retry the operation+
+Command Timeout on Disk [5001086000000f27 0]+
+Resetting Disk [5001086000000f27 0]+
+Error on 5001086000000f27 0, Op: 0x2A, LBA: 0x370fa800+
+SenseKey: 0x0B, ASC: 0x00, ASCQ: 0x00 after 1 Retries+
+Retry the operation+
Ready.

At this stage it looked like a geniune HEAT issue with the ATTO EXPRESS R380. Many forums indicated this ATTO R380 card had issues when running too hot.

OK so then I looked at its presence in the x8 PCIE slot #4 in my MACPRO 2007 model. YEs I realise it sits directly under internal DDM's tray#3 adn ddm tray #4. So I decided to move this card to a lower slot (SLOT 3) and also perform a CLEAN OUT THE DUST of my mac pro. (after many cans of compressed air and cool YOUTUBE video on how to pull all the fans etc out). I reseated all the cards and fired up the MACPRO 8core.

I used configuration setting config02 of the /system/library/CoreServices/Expansion Slot Utility.app (which btw seems broken at OSX 10.5.6) to:
• set the ATTO EXPRESS SAS R380 card into a x4 PCIE slot which I used as SLOT3 in this MACPRO. I did this to at least provide some physical distance between the HBA and the internal DISK trays #3 & #4 which ran hot with spinning WD1002FYBS 1TB disks in them. NOte that having the R380 RAID HBA in a x4 slot is fine. There's no way I could ever get that data rate due to a full x8 lanes anyway from his 2007 MACPRO 8core.

I conducted the same COPY OPERATION tests and although having far less FANcontroller (nill | zero | ziltch | none) messages and few OFFLINE situations again I still encountered HFS+extended journalling errors.

I then archived the material overtime to an Ultrium4 tape drive (4 LTO tapes) and decided to REMOVE the HFS+ journalling.

I restarted the MACPRO and power on recycled the disk array so that all disks in the external PROAVIO EB8MS JOB enclosure were seen by the ATTO CONFIGURATION TOOL.

I reinstated the content from my archive onto one of the RAID0 arrays and initiated my copy processes again.

Some success with ATTO EXPRESSSAS PCIE R380 in SLOT3 (x4 PCIE) and HFS extended with no journalling and data rates as high at 380MB/sec and (450 MB/sec with AJA system tester). Hmm!

HOwever after some time I saw yet another ATTO EXPRESS SAS ERROR error that caused one of the arrays to be offlined once again.

THis was a FILE SYSTEM error of 'overlapping allocation extents' (Wah! something was hosed) . THis brought up by the DISK UTILITY.app from a first aid/VERIFY operation.

I fixed this with Tech Tool PRO4 (ttp4) and looked at this error closely.

I then deleted all the RAID groups (recall I had an archive of this material on ULTRIUM LTO4) and built the raid groups as follows with the ATTO CONFIGUATION TOOL:
• RAIDSET01: 4 x 1TB, RAID0, 1MB interleave, 512 sector size (not 4K!) , READ ACCESS = ALWAYS, PREFETCH=3* (tried 6 too = OK)!
• RAIDSET02: 4 x 1TB, RAID0, 1MB interleave, 512 sector size (not 4K!) , READ ACCESS = ALWAYS, PREFETCH=3* (tried 6 too = OK)!

THe above configuration seetings work well.

So to summarise, I used *these settings* for the MACPRO 8-core 2007 model + PROAVIO EB8MS JBOD enclosure with 8 x 1TB WD1002FYBS DDMs + ATTOTECH EXPRESSAS R380 HBA:

• /system/library/CoreServices/Expansion Slot Utility.app configuration setting config02 (the 2nd one on the list from the top) with PCIE slot3 and PCIE slot4 as PCIE x4 lanes (slot2 is x1 and slot 1 is x16 for the GPU).
• *ATTOTECH EXPRESSAS R380 HBA is MACRO PCIE SLOT3*
• ATTO CONFIG TOOL RAID0 group configurations: RAID0, 1MB interleave, *512 sector size* (not 4K!) , READ ACCESS = ALWAYS, PREFETCH=3
• *disable HFS+ Journalling on the RAID0 volumes*
• for extra performance OMIT these logical volumes from TIME MACHINE and SPOTLIGHT.

Notes:
• sure!! RAID3, 1 or RAID5 would have helpd me out *MAYBE if* and only IF one DDM was affected. THis case RAID availability was of no use. Thankfully I use LTO4 UTRIUM tape drives for archive. THis saved me yet again. 🙂 (A lesson for some!)
• at OSX 10.5.6 /system/library/CoreServices/Expansion Slot Utility appears to be broken. If you have a later 2008 MACPRO these use PCIE2 slots and have automatic configuration so don't worry about this. Sadly I have a MACPRO 2007 8 core.
• system is very stable now and no more fancontroll messages either.

hope that helps someone.

W
HK