Optimizing multiple simultaneous finder copy jobs

Question

Optimizing multiple simultaneous finder copy jobs

This is a question about how Leopard (or any Mac OS for that matter), performs multiple simultaneous instances of copying finder items. I'm in the process of consolidating photo data which resides on multiple partitions on different external drives onto one single 500gb firewire drive. If I drag one folder of 20gb of images onto my 500gb drive icon, then drag another folder of 20gb of images, then another, etc., what is the OS doing for the copy? What I really want to know is this: does the OS "partition" where the copies are stored on the target? For example if the 500 gb drive is empty (0), does the first copy get allocated space 0-20, the second copy get allocated 21-40, and so on? Or does the multiple copy just lay down wherever it can, potentially fragmenting my files all over the target drive?

If it does fragment, which I'm guessing might be the case, is there an easy way to queue the file copies so that they're done in succession so none of the files are fragmented? Automator would seem a likely candidate.

Also, if the Finder is smart about copying and makes every attempt not to fragment files, does performing multiple copies at the same time slow down performance (drive thrashing, etc.), as compared to successively copying each job after each is complete?

I've never seen any discussion about this topic and thought it might be useful for everyone to know. BTW, my files within each folder vary from a few megabytes to around 250 Mb in size.

Mac Pro 2.66 quad, Mac OS X (10.5)

Posted on Nov 1, 2007 4:15 AM

Reply

Answer 1

R C-R

Level 6

17,794 points

Nov 1, 2007 4:54 AM in response to Paul Takeuchi1

Or does the multiple copy just lay down wherever it can, potentially fragmenting my files all over the target drive?

OS X attempts not to fragment files by avoiding reusing spaces between files. Although it won't answer all your questions, you should find About disk optimization with Mac OS X an informative read.

Reply

Answer 2

The hatter

Level 9

61,031 points

Nov 1, 2007 4:54 AM in response to Paul Takeuchi1

FireWire having to share its single channel. FW400 vs 800
The speed of the drives
The Finder and programs can cache files to memory before writing.

4GB file(s) and larger - with small files even 1GB they should fly. With 4GB you will see a slowdown.

If you have 20GB, let that run by itself. And don't 'overwhelm' your FW drive (and USB would be even worse).

yes they will be fragmented, which any modern drive can deal with. Use a stop watch - I think most people have an idea of what their system can do.

I would copy to an internal drive, maybe in unique folders, then merge those when you consolidate to their final Firewire drive - you could use a program like SuperDuper that copies fragmented files to single "fragment"

Some people use RAID so that even copying large files won't affect or impact the system and go quicker.

Reply

Answer 3

R C-R

Level 6

17,794 points

Nov 1, 2007 5:36 AM in response to The hatter

The hatter wrote:
I would copy to an internal drive, maybe in unique folders, then merge those when you consolidate to their final Firewire drive

I disagree. Modern drives are much more sophisticated than most people think. They use out-of-order execution to speed up operations by taking advantage of the head's current track position vs. the next one it should go to. So even if the OS delivers file A part 1, then file B part 1, then file A part 2, etc. to the drive's write buffers, it is still capable of writing the two File A parts from its buffers to adjacent sectors before the head moves to where it must write the File B sectors. (This is one reason drive benchmarking utilities offer the option of turning off buffers to measure the "raw" speed of the drive.)

Besides, OS X is a multi-threaded, multitasking OS & it is quite likely the OS will be paging between memory & drive storage during long Finder file copies anyway, so the drive isn't really servicing just that file copy request on an internal (startup) drive.

yes they will be fragmented, which any modern drive can deal with. Use a stop watch - I think most people have an idea of what their system can do.

No, they probably won't be fragmented unless the destination drive is fairly full (see the tech topic mentioned in my other post) & yes, do use a stopwatch to decide what works best for you. Just be aware that in light of the above, results will vary depending on several factors more complex than the simple sequential write model often used to explain how disk access works.

Reply

Answer 4

Nov 1, 2007 1:57 PM in response to R C-R

{quote:title=R C-R wrote:} Modern drives are much more sophisticated than most people think. They use out-of-order execution to speed up operations by taking advantage of the head's current track position vs. the next one it should go to. So even if the OS delivers file A part 1, then file B part 1, then file A part 2, etc. to the drive's write buffers, it is still capable of writing the two File A parts from its buffers to adjacent sectors before the head moves to where it must write the File B sectors.{quote}

RCR, you make some good points and I appreciate the link to Apple's note about fragmentation. I agree that the drives are probably more sophisticated than we think.

This is an interesting problem, more critical perhaps for video people than others because they have huge data streams that have to be read and written in real time.

After further reflection (and a read of Apple's disk optimization article), I believe that it's very hard for the Finder to determine what is efficient. Sure writing data sequentially onto a new data-only disk will speed reading of that file later, but as soon as you edit it and resave it to that drive that's now more full, you will lose that benefit as the drive heads try to find free space. If it takes me ten minutes to queue up a multiple copy job now (vs. 20 seconds just dragging folder after folder after folder), do I save that time later when I'm using my data? Unlikely for most types of files, I'd bet (except video).

So perhaps I'm being too fussy and it's just better to drag folders however's convenient and let the randomness of how they fall on the target drive be okay. Save time now vs. later.

Still it would be nice to know how the Finder tries to feed the data to the drive. It must be using some kind of algorithm which tries to please the user by finishing quickly--or at least appearing to. I know that if I copy a folder full of 100 files, then a folder full of three files, the second folder will start off slow then finish quickly, way ahead of the first folder. Without my telling it, the Finder has no idea that I don't care which job gets done first as long as the files are copied sequentially. Though I feel like I've wasted enough time on this already, I think it would be nice for the finder to offer a dialogue which offered the choice of sequential versus parallel copying.

Reply

Answer 5

Nov 7, 2007 12:03 PM in response to Paul Takeuchi1

I actually posted my question on macgurus and there were many elucidating and thoughtful replies there. Go here for the discussion:

http://www.macgurus.com/forums/showthread.php?p=113160#post113160

Paul

Reply