Slow Copy Between External Drives

FW ports are rated in terms of instantaneous interface speed, not average throughput. A FW400 port has a theoretical maximum interface speed of 400 Mbps or 50 MBps. But the average throughput for two similar FW400 drives connected to the computer will rarely exceed 20 GB per hour or about 6 MBps. FW800 may reach 32 GB per hour.

The throughput will also be affected by how many files and their sizes. Transferring 10 MB made of of 10,000 1 KB files will take longer than transferring one file that's 10 MBs in size.

This does not even consider the drives involved, the case in which they are installed, the quality of the cables, and the bridge firmware used in the enclosures. All of these are potential factors that will affect average throughput.

So your drives aren't really slow, rather your expectations are too high.

I've been wondering about a similar issue with file copies. I understand that copying many small files vs one large file does entail overhead for opening and closing files. My ad hoc opinion, though, with Snow Leopard is that copying files takes far longer than I'd expect. For example, I've copied a large multi-GB file and achieved transfer rates of nearly 80MB/s with FW800 7200rpm drives (from FW drive to my internal 7200rpm drive). But, when I do a copy of many small (<1KB) files my average transfer rate goes down to about 1.5MB/s. In this case, I'm doing the copy entirely on my internal drive. I use iStat for the performance numbers.

I started getting concerned when my 3.06GHz MBP started getting more and more spinning beach ball waits, especially when swapping between applications; in particular to Java-based apps. When the beach ball appears there is typically a lot of disk activity but low average throughput. It's almost as if Snow Leopard is doing synchronous IO--not suggesting that it does, just that this is what it seems like. Basically, it appears like Snow Leopard does not like handling large numbers of small files. BTW, I have about 30% free space on my 500GB hard drive and lots of available RAM.

I'm going to do another Repair Disk (after booting from my installer) and see if that helps. In the past it seems like resetting the SMC has helped but since that also entailed a reboot there could be other factors in play.

Any suggestions on under-the-hood optimizations that might be relevant?

Thanks,
Kevin

Kappy;

Very interested to hear hat you said about expected speeds over FW400 (20GB/hour). However, see if you can shed some of your light and wisdom on this confusing issue:

1 ~ External Drive A (FW400) has 2 partitions and one of those holds 136 GB of data. I want to reformat the drive as a single partition, so I copy the data to external Drive B (FW800). It takes about 2.5 hours to complete the transfer.

2 ~ I reformat Drive A to a single partition.

3 ~ I copy the data back from Drive B to Drive A. Except that this time it's 11 hours later and it still has another 16 minutes to go.

Same drives, just a repartition performed on one. What could be the cause of the dramatic slowdown in the copying speed during the 'copying back' stage? Should I be concerned that the partitioning was not correctly applied and start all over again?

The more FW devices you have daisy-chained the slower transfers will be. Because there is more inter-device arbitration going on as you add more devices.

Also, if you have even one FW400 device connected to a chain of FW800 drives, the entire bus will be slowed to FW400. Even slower if you have a camcorder or other miscellaneous FW device daisy-chained. miniDV camcorders are notorious for only supporting 'FW100.'

Exactly what drives are they, what is their capacity. How full was the FW800 drive after you copied your 136GB to it? Exactly how do you have them connected to your Mac. Did you change the connection(s) between the two copy operations? Did you restart your Mac between the two copy operations? What partition scheme & format does each drive have (before & after)?

Thanks for responding MartinR.

Drive A (FW400) is an Iomega 750 GB — the ones that are styled to fit in with Mac Minis. It provides several extra USB and FW400 ports, and so I do use it as a USB hub.

Drive B (FW800) is a brand new Western Digital MyBook Studio Edition II (2TB), partitioned into 4 volumes. The volume that I copied the data into temporarily is 25o GB capacity and was empty beforehand. So it ended up a smidge over half full.

*How they are connected:*

Drive B is connected directly to the Macbook Pro's FW800 port. No daisy chaining of any sort, it's the only device connected to that port.

Drive A is connected to the hub built into my 30" Apple Cinema Display. It is connected by both the FW and USB connections (in order to feed its hub functionality), but a dip switch on the back of the drive should force it to use FW by preference in matter of general data throughput. A couple of other devices are further connected to Drive A's spare ports; these are things like my iPhone charger/connector lead and a colour calibration sensor. So the chain looks something like:
Mac > CinemaDisplay > Drive A > minor peripherals

The biggest 'odd' factor here for me is that No, I have not changed this arrangement since the initial copy from Drive A to Drive B was made. Everything is exactly as it was then.

Nor have I restarted the Mac between copy operations. This also means that I have not restarted it since repartitioning Drive A. I don't think I have ejected and remounted it either. Maybe either of those things would help. I'll try them. In the meantime, does any of the above raise any red flags with you? For the record, I never expected to get FW800 speed in any part of this operation; clearly the copying can only take place at the speed of the slowest drive, which is FW400. But 11 hours to copy 136 GB from B -> A seems awfully slow, especially since the original A -> B copy took a fraction of that time.

Thanks in advance for any insight you can offer.
Rick

Oh, Partition schemes:

All drives are currently formatted/partitioned to GUID partition table.
AFAIK Drive A was also partitioned this way prior to repartitioning, but of course I have no way to verify that at this stage.

Um, drives that have both USB & FW connectors should never have both connected at the same time. What position was the FW switch in? Auto or 1394?

I am thinking there may have been some directional sensitivity, since the Cinema Display has built in FW and USB hubs and your drive is unusual in its apparent support for FW & USB connections at the same time. Are you sure the switch was not on Auto?

What I mean is, the first copy action was Iomega FW400/USB drive -> CinemaDisplay FW/USB hubs -> MBP -> WD FW800 drive. The FW400/USB drive would have been the initiator and it could have been either a FW or USB transfer. When you did the reverse copy, perhaps your MBP and/or your CinemaDisplay had problems figuring out whether to transfer via USB or FW since both were connected to your FW400/USB drive. Particularly if the drive switch was Auto. Just guessing at this point.

Thanks again for your reply.

The Iomega Minimax drive seems to be unusual in that it does permit both USB and FW connections. The switch on the back makes this possible; if set to Auto it creates a data connection using whichever type is connected first. If set to 1394a it will always default to a Firewire connection if such is present, even if the USB cable was connected first (in such a case the USB data connection is closed and the FW one opened). However, the USB connection still remains in order to act as a hub, whilst data is routed through the FW port. It took some rummaging through the Iomega site forums to confirm this (and conflicting information does exist) but this what I believe to be the case. Indeed, if it were not the case and simultaneously connecting USB and FW was unsupported, then what would be the point of the switch?

(Yes, my drive is definitely set to 1394a, not Auto).

On the other hand, this is definitely an atypical feature and you may be right in your supposition that the the Mac's data-routing controller got confused, especially if the drive's FW/USB resolution was imperfectly implemented.

For now all my big data moves are done, and everything from here on will be incremental unless there is a need to do a big rearranging of files for some reason, so I'm not going to worry about it too much any more. If I need to move 100+ GB again and the problem reoccurs, then I'll revisit the issue.

Thanks again for your time and input.

Message was edited by: Rick Lecoat