I spoke too soon. It *seems* like as soon as I kicked off another Finder copy in parallel with the monster one which was taking place, I did start to see I/O errors.
[tons of them]
.
.
.
Apr 24 23:12:21 EmuraPrime kernel[0]: disk5s2: I/O error.
Apr 24 23:12:27 EmuraPrime kernel[0]: disk5s2: I/O error.
Apr 24 23:12:28 EmuraPrime kernel[0]: disk5s2: I/O error.
[EmuraPrime:~] emura% grep "disk5s2: I/O error" /var/log/kernel.log | wc -l
1005
[EmuraPrime:~] root# fsck_hfs -fn -c2g /dev/rdisk5s2
** /dev/rdisk5s2 (NO WRITE)
Executing fsck_hfs (version diskdev_cmds-491.6~3).
** Checking Journaled HFS Plus volume.
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume DroboS appears to be OK.
So most metadata is okay (i.e. as much as fsck_hfs can confirm).
And while this DroboS continues to wildly spit out I/O errors in the log more than once a second, a check of some of the file data that was being copied at the time of the previous I/O errors is okay too:
[EmuraPrime:/Volumes/DroboS/Chris] emura% paranoia_verify_checksums.zsh *
Total_Files = 340
Mismatched_Checksums = 0
Matched_Checksums = 340
Unrecognized_Extensions = 0
Checksums_Not_Found = 0
( these are just raw Nikon NEF photo files where I keep the md5 embedded in the file name itself )
Also, doing I/O via dd(1) from the raw device is interesting depending on which size I use for the I/Os vs. hitting errors vs. performance.
Small I/Os (512, 4096, etc.) seem to translate into really bad performance (5MB/second), but no errors.
Larger I/Os get throughput up into the 30MB/second range, but I/O errors from time to time.
Much larger I/Os and we bail with I/O errors which make it back to user-land and dd(1) will bail.
Though throughout, looking at the I/O trace, it's clear that the Drobo has internal smarts and is "always trying to do something clever" on its end. Good stuff. Fun for debugging and troubleshooting. Looking forward to seeing what their technical support folks come up with.