Very large collection, slow iTunes

Is your music on an external hard drive?

Macbook Core 2 duo (Black) Mac OS X (10.4.8)

How much memory do you have? My music collection is approximately half the size of yours--35,000 songs/250 gb. When I first installed iTunes 7 I found a lot of stuttering and pausing when I searched using album view. I upgraded my RAM from 1 gb. to 3 gb. and now things work smoothly. Perhaps more RAM will help. I also think with your large collection, processor speed and hard drive speed/buffer size make a difference. Obviously, the faster the better.

I actually might make sense to use parallel to run a media manager designed for windows. I personally LOVE mexp, which is a library manager that works with winamp. It handles my 100GB collection like it's nothing. I recently got a Macbook, and I am thinking of using parallels JUST to use MEXP. Here's a link if you are interested:

http://www.mexp.dk/index.php

If you have a lot of smart playlists, this may slow things down immeasurably. I'm on an MDD G4 with 1.5GB of ram and about 14k songs in my itunes library, and I was experiencing massive slowdowns until I unchecked 'live updating' in most of my smart playlists and started limiting them to a few hundred songs at most.

So far, so good. No more beachballs or 10-second lags between click and response. This seems particular to itunes 7 (currently on 7.02).

I'm desperately trying to solve this problem as well. Here's my experience:

- almost 60,000 songs/300+GB
- Dual 2.0 G5 w/ 3.5 GB of RAM

- takes time to load library at application start-up (understandable and doesn't bother me).

- have run it on both internal and external firewire 400 drives with same problems.

- my main problem is getting the beachball when I <get info> to alter track info. There is some lag time when using the search box as well amongst a lot of other things. In general, the application just doesn't respond as fast as I would like and it seems to be the size of the library.

- I added 2GB of ram (up from 1.5) and that didn't help.

- My percentage of HD used does not seem to affect it.

My only other option (that I can think of) is to get a faster HD (or H Drives and stripe them) to try to increase access or seek times.

I found this bit of info though in another thread that I found very interesting...
_____________________________________________________________
"Re: Need Help with EXTREMELY Large Library!
Posted: Sep 7, 2006 7:21 AM in response to: Bill Ryan2

I don't know how much of the slowdown is iTunes managing its playlists, and how much is the operating system handling the iTunes Library (the single file containing the tag information).

I ran Sonar ( http://www.matterform.com/macsoftware/filesecurity/) to watch file read/write activity while editing track tags. I saw a lot of iTunes Library temp and XML file writing.

My iTunes Library (for 160 Gb) of music is 53 Mb. But the Spinning Beachball corresponds to writing this file. Maybe a faster disk would help things."
_____________________________________________________________


This is my fear...that the fact that iTunes stores all that good info in one file (terrifying to me), that it can only handle so much.

Has anyone tried any sort of RAID array with fast HD's with large buffers? I'd love to know if this will help before I sink a bunch of cash into this setup. Multiple libraries are not an option for me...it's kind of like keeping half my records at another house.



Dual 2.0 G5 w/ 3.5 GB of RAM Mac OS X (10.4.8)

Dual 2.0 G5 Mac OS X (10.4.8)

Dual 2.0 G5 Mac OS X (10.4.8)

David, my problem almost duplicates yours almost exactly, although I only have 2GB's of RAM. I was just looking into getting 2 more gigs of RAM in hopes that it would help the iTunes Beach-Ball Syndrome :/ I also moved my iTunes music folder to an external drive to see if that would help in any way, but no 😟

I do not have an answer about a raid system helping, but I am afraid that it is our many playlists that slow it down and cause the beach-ball every time we make a move in iTunes. iTunes 6 did not care how many playlists you had, but iTunes 7 does not like them...which, to me, is a complete drag. I like having a playlist for each CD I put into it. I have read and been told, that this is now a no-no :/ Going back to iTunes 6 is not an option either, since all the playlists added after I upgraded to 7 will be lost.

My iTunes is an integral part of my Mac usage. I listen to it a lot and this problem has really, really frustrated me.

You put the problem into words much better then I could have. I just really hope that in one of these updates Apple will fix this problem, or maybe it will be fixed in Leopard???

I guess we can only hope...

G5 2x2 / Pismo G3 400 / iBook G4 933 Mac OS X (10.4.8)

I just read your post in that other thread and replied there as well (it's a very helpful thread by the way): http://discussions.apple.com/thread.jspa?messageID=3996066&#3996066

FWIW, doubling my RAM didn't help. It might help when you are in album view, but I still got the beachball while editing tag info.

I'm going to try to trim my playlists down because I don't need them all. Many of them are mixes I started and abandoned after a few songs, or burn playlists. I don't really use smart playlists except for "recently played" and "top 25."

What I'm learning is that the problem is iTunes and the way iT 7 stores and manages the info. Apple will have to respond to this, but we'll have to wait in the mean time.

For me, I'm going to try a few hardware upgrades and tweeks, but like most people here, I don't want to change the way I listen to or archive my musical collection and my library will continue to grow. It's not unusable by any means, it's just frustrating to have such a powerful computer and OS and to still be waiting to change the name of a **** band to keep things tidy and organized. Unless someone convinces me otherwise, I'm coming to the conclusion that the software is the bottleneck. That guy's test where he monitored the file writing activity combined with the fact that others have mentioned that stepping up to a Raptor drive and only getting limited improvements have got me thinking that it's iTunes.

I have Traktor (multiple player DJ software), but haven't hooked up my Mbox yet so I don't know if that will help. But even if Traktor responded quicker to playing the tracks, I still can't organize and edit with it like I use iTunes to do.


Dual 2.0 G5 Mac OS X (10.4.8)

Folks

The problem is that these 150 GB + libraries are much, much larger than anything considered when iTunes was developed. And still is. The slowness is reading and writing to the database file. Every change means rewriting the entire file - quite fast for a 2 gig library, interminable for a really big library. (Consider the time it takes to open a 50 - 60 MB Photoshop file, or to save it, that's what you're doing when you read or write to the db file in iTunes.)

Yes a faster disk will make a difference, as will more RAM, but not that much. The simple fact of the matter is that iTunes doesn't scale well, and to make it scale well you'll need to re-design it to take account of really large libraries.

The work arounds are there: Multiple libraries. Hold down the option (or alt) key when launching and create a new library. How you split them up - by genre, by artist, whatever - is up to you. You can only have one library open at a time, which can be a pain. But do you really need all 40,000 tracks available, all the time? It's a work around, not a solution, but until they remake iTunes, it might be the best solution.

Another option is to use different software: Songbird is an open source app that is under active development. It's got iPod support now. I'm not sure what its' support for really large libraries is, but you can email them and see if they can build it in.

Finally, to the poster above who is terrified of losing the database file: Back It Up. Before you make changes, just simply duplicate it in the Finder (File -> Duplicate). Then do the same after. If the db file goes pear shaped, simply replace it with the copy.

This problem will not be rectified in Leopard, it's not an OS problem. Give as much Feedback as you can to Apple, maybe they'll redesign it for iTunes 8.

Regards

TD

Folks

The problem is that these 150 GB + libraries are
much, much larger than anything considered when
iTunes was developed. And still is. The slowness is
reading and writing to the database file. Every
change means rewriting the entire file - quite fast
for a 2 gig library, interminable for a really big
library. (Consider the time it takes to open a 50 -
60 MB Photoshop file, or to save it, that's what
you're doing when you read or write to the db file in
iTunes.)

Hi Terence,

Thank you for trying to help 🙂

I understand what you are saying here, but you are overlooking the fact that I had a 200GB +- library in iTunes 6 and it worked flawlessly! This problem started right when I installed iTunes 7. So, something changed in iTunes 7 to cause this problem. I would have reverted to 6 immediately, but I was sure it would be fixed in an update....wrong!

I know I can have multiple libraries...I do not want them though :/

I know I can try other software...I have an Apple, who also happen to make iTunes, so I would have hoped that they would have thought of the fact that people will load their CD's in iTunes and they will end up with huge libraries.

I love Apple and I love my Mac, but this problem really makes me angry...

David, I hope the Raptor, or whatever you get helps, but I am afraid it will all come back to iTunes making that beach-ball spin :/

G5 2x2 / Pismo G3 400 / iBook G4 933 Mac OS X (10.4.8)

Thanks for more clarification. I only started looking in earnest tonight for some answers on ye olde internet, but it took a good 6 hours to stumble across this thread that answered so many of my questions so well.

Fact is, I actually do need all 50,000 files available when DJ'ing or sampling/reworking music. Splitting up my library would be like only being able to flip through half your record collection and multiple libraries would be extremely frustrating if the creative urge were to strike and I didn't have the goods on hand. And what about the drunken requests for some obscure song that I usually don't listen to? I needs all my music the same way I hauled around crates and crates of records.

Songbird would be exciting if it imported your iTunes library and all of your data. A key component for organizing for me is rating the songs. If I lose those ratings, I won't remember what the one song I loved off that album that I only listened to once a year ago was. It's the way it goes with a large library.

As far as the copying of the iTunes library file, I do do that, but it's still a little dicey. Several times I've found myself with iTunes not being able to associate every file after changing a HD or something. Some of it was operator error, and sometimes it was just Gremlins...but nonetheless, it's keeping all my eggs in one basket so to speak and that makes me nervous. It's also iTunes' achilles heal as far as the speed issues that we're talking about. I was going to try to fit a 3rd cliche in there, but it's late.

Thanks for the info Terence.

Dual 2.0 G5 Mac OS X (10.4.8)

Hey Terence:
First off thanks for such a detailed post but I have to disagree with you your opening sentence. Wasn't iTunes 7 developed with television shows and movies in mind in addition to music? A one-stop media center (hate to use those words) for all of your stuff? And if so, that the size of the libraries isn't that weird right? Video takes up much more space that .mp3 files, Apple aggressively markets iTunes as THE media player, a little 100 GB library like mine isn't all that unusual anymore and as one of the previous posters mentioned, iTunes 6 was fast enough seemingly to handle things.
The idea that Apple would release something an application unable to handle the above amazes me (good days) and really irritates me (bad days) and I for one think it's just crappy software right now. I'm holding out hope that iLife '07 was delayed due to an iTunes re-write so maybe it's on the way.
As for Songbird, it's okay but pretty slow and light-years away from a v.1 release...

W. David

A television show is a single entry in the db. An album will be 12 to 15 entries, possibly more. A hundred gigs of music will lead to a much bigger db file than the same amount of 60 minute television shows. The slowness is caused by the reading and writing to the disk. The size of the video file versus the size of the mp3 file is irrelevant to the size of the db, what counts is the amount of entries it creates. The files are stored in the Finder, not in the db.

Secondly, I do think that a 100 gig library like yours (and indeed mine) is still unusual. Apple has sold 90 million iPods. A lot, a very great lot of them are Nanos and Shuffles. I know maybe 15 people who have an iPod and every single one of them (except me) can fit their entire library on it. When you consider that amount of iPods, I wonder what the average size of library is? Bet the bell shaped curve doesn't apply, and that it's heavily skewed towards the smaller end.

Finally, my point about Songbird is not that it's the finished product, but quite the opposite. It's still at a point where you might be able to influence it's development.

Regards

TD

I have to disagree with your idea of small music libraries - though my family is music oriented -- Myself, Mom, Dad, brother and one of my sisters each have >100gb libraries. (ages 60, 57, 35, 31, 26)

I have over 1300 CDs and 200 LPs coverted in full AIFF size. My large library is a drag even on my MacBook Pro Core 2 Duo, 17" 2.33 GHz, 2gb with 1 Tb external drive (7200 rpm) via Firewire 400 or 800.