My observasions regarging this problem.
I have 16GB ram i7 3.5Ghz, 256GB SSD boot drive and a fusion data drive.
I do alot of big file copies, and lots of smaller file copies, i.e. a massive photo library and lots of DVDs that I regulary rip. And lost of music files as well, e.g. CDs and my own recordings.
This causes the file cache to grow in size.
What I see is the following.
After alot of file copies (they have completed at this pont)
a) the com.apple.IconService goes a bit mad and consumes about 300mb of memory.
b) Spotlight indexer also goes a bit mad.
c) the File Cache in creases so that all of the available memory is used.
At this point Mavericks obviously flushes some of the file cache to disk (swap), alas what it does seem to do is max out the IO channel. So if you happen to be listening to a track or watching a video, it freezes whilst the disk goes through a massive write process (disk light goes bright red for upto 15 seconds).
Now the more Ram you have the more it seems to flush in one go.
Also if you happen to have an SSD (which will max out a SATA 3 channel) all of the available bandwidth will be used, causing everything else to slow down.
I've yet to try this (it will take a long time), if the swap disk were a regular spinning disk, the IO channel will not be maxed out, leaving enough bandwidth available for other apps if they are running off a different disk.
Being unix, there must be a way of relocating the swap file elsewhere so see if this fixes the problem (ok is will cause more of a problem when RAM is really exhausted).
I recon the fix will be to be less aggressive with the memory purge to disk.
Unfortunatly I tried monitoring with fs_usage, but that hangs during the massive disk usage (as does activity monitor) so does not acutally register what is causing the IO. So the above is just my eduacated guess.
I do not see the same behaviour on my wifes imac with only 4GB memory and a slow spinning disk. Her machine runs slower on average but does not experience the massive slow downs my mac experiences.
Note I also run boot camp with windows 7 and get now slow downs at all.