As you can see from the very good comments, there are a lot of possible variables at play here. Let me offer a benchmark:
My main computer is a 2007 MacPro with 21 GB of RAM, the 5770 video card, and a bunch of conventional HD.
As Alan has pointed out, the first thing is to be sure you have as much RAM as possible, the fastest possible HD, and keep your primary HD, the one that contains your OS/APP/Library as empty as possible.
As your first HD passes 50%, consider off loading your Master images to another HD. Referenced Masters are not, in and of themselves faster than a Managed Library, but if your Managed Library pushes your first HD much past 50% full, then your computer is going to spend a lot of time looking for free space for work/scratch files and for space to rewrite your Versions as you made Adjustments.
Smaller/lower quality Previews help scrolling speed, but probably don't help much when you actually pull up a single image. I am not sure if Aperture uses the Preview or the Master for the first view of an image, but I know that the Master is read when you view an image at full resolution.
Pulling up an image requires several steps which engage all parts of your computer:
-- The HD reads the Master file.
-- Then the Version file is read.
-- Then the two are combined and the GPU puts the resulting image on the screen.
This is a lot of computing. On my computer it can easily take 3-5 wall clock seconds from the time I click a 120 MB TIFF in Browser view until the image is on screen. Zooming to full resolution can take the same amount of time. But since I have ample RAM, this only happens once. Thereafter, everything is in RAM as shifting views, etc., is instantaneous. And, as I have a lot of RAM, I can read as many as ten images into RAM and then step through them, back and forth, without any "Loading" message reappearing. I can even do up to about five images at full res without seeing a "Loading" message.
But if I unleash a computationally heavy Adjustment, like Shadows or Edge Sharpen, then there will be lags as the entire image is rerendered and the Version is rewritten.
Obviously, a newer MacPro or one of the new i7 iMacs would probably be faster than my machine, but the 5770 video card is Apple's current release.
I hope this slightly geeky post helps you determine exactly where your bottle necks are.
--
DiploStrat 😉