Your screen resolution isn't 600dpi (it's probably around 100 dpi), so your scans are probably overkill, and likely to a) slow down iTunes and b) increase the size of your library. A CD insert is about 4.75" square, so your scanned images are presumably about 3150x3150 pixels. In other words, the cover art would have to be over 30" wide on your screen before you began noticing pixelation.
Most artwork you download from iTunes is 600x600 pixels, which displays very nicely. Scanning at 150dpi will get you art that's a little bigger than that. If you then compress the 150dpi images using a medium-to-high JPEG quality setting, they should appear much like Apple's downloaded artwork.
How big are your image files, in MB? Perhaps the display problems you're seeing come from iTunes trying and failing to display or downsample huge files.