Very large collection, slow iTunes

Terence, you still do not mention that iTunes 6 worked fine with my large library and it seems that it worked with others very large libraries as well.

It was upgrading to iTunes 7 that has caused all the problems.

My question, which an Apple soft developer must know, is What Changed to make it not read the xml file as well as 6?

Yes, I know you are not an Apple dev, well at least I think you are not LOL But I do hope they check this board every so often 😉

I appreciate you taking the time to try to answer our call for help.

First off thanks for the considered response. Now back to front...
So Songbird we might be able to influence but no on iTunes? You don't think it might be good for Apple to take a look at users experience and maybe make an effort to understand how everyone is using the software? And the developers of Songbird seem to be focused more on the GUI that the coding if their development cycle is any indication.
Secondly it might be that 100+ GB liraries are not the norm but how do you quantify that? Is it 10% of the user base and if so, how many users would that be? I think a lot.
Thirdly number of entries versus size of the individual entries. You make a great point there (1 movie=1 entry no matter what the size) which would make a whole lot of sense when it applies to the problem of playlists. Would they then count as another entry? Album AND playlist if you get me?
Lastly what about the perfomance differences between iTunes 6 and iTunes 7? What do you think happened there? iTunes 6 was fine where 7 is a beach or at least a beachball...
Thanks again most sincerely for the response; I'd love to think that iTunes 8 or 7.5 or whatever will take the perfomance back to where it was and really do think the idea of a ProTunes would be worth considering.
Cheers and I'm off to walk the dog!

W David

First off, I'm not an Apple Developer, never have been, and unless there's a radical - and very unlikely - change of direction in my future, never will be.

Secondly, I'm not defending iTunes, it's not my job or my intention. When I wrote I wrote only to offer affirmation to those who are having issues, plus to point out the work-arounds, plus the possible plans of actions. I wrote to say "Your iTunes is not broken, but rather, it's pushing at the boundaries of what's possible with iTunes. Therefore, there's no 'fix' like trashing a pref file will sort this one out. It's a built-in issue and will have to be a coded solution."

So Songbird we might be able to influence but no on iTunes?

Actually, if you re-read the post where I linked to Songbird, I also linked to the iTunes feedback page, and said Give as much Feedback as you can to Apple, maybe they'll redesign it for iTunes 8. So, move forward on both fronts. However, Songbird is Open Source, and therefore driven by community demand. Apple is a lovely company I'm sure, but iTunes is developed for Profit. When there's a critical mass of very large libraries, or when they see it coming, Apple will focus on the issue.

Secondly it might be that 100+ GB liraries are not the norm but how do you quantify that? Is it 10% of the user base and if so, how many users would that be? I think a lot.

I can't quantify that, only Apple might be able to. However, I suspect there are a lot of very large libraries, but not as many as we might like to think. An Apple executive was presenting at an event recently, where he mentioned - inter alia - that Apple have sold 90 million iPods. Let's say there's another 10 Million iTunes users out there who don't use iPods - if only to make the Math easier. A million very large libraries is only 1% of the User Base. How many do you think there are? And what proportion of the User Base would need to be having problems before it becomes critical to re-develop iTunes? 1%? 10%? 20%? Eventually it will, but eventually could be a long time. Hence my suggestion about Feedback.

Yes the database tracks the songs plus the uses they are put to - i.e playlists etc, absolutely and associated Metadata too, like playcounts, ratings etc. Obviously the more complex the arrangements and metadata, the more entries in the library, the bigger the file, the longer it takes to read and write to disk. (And btw the xml file is the read by apps that use iTunes media like iPhoto or iDVD, the library file is the other one..)

As to what happened between 6 and 7? I would have thought that was obvious: a massive increase in metadata: Gapless playback and Coverflow. You have no choice about the first - each track is assessed for Gapless Playback - and whether or no you use Coverflow, it still polls for it in it's operation. I did a quick comparison between by v6 library (stored in the Previous libraries folder and my v7 one. My v7 is about a third larger, and the library has not grown by 33% in that time, nowhere near.

More and More people will develop larger and larger libraries. So this will have to be dealt with eventually, but that could be sometime away. Therefore, I suggest, the right thing to do is to Feedback to Apple, plus have an eye on alternatives in case Apple are too slow to respond.

Finally, I'm quite lucky, my 110 gig library opens in about 55 seconds, and I have no significant lag when manipulating tracks or metadata. However, my use of that library is possibly quite unusual. I don't rip on that iTunes, don't tag on that iTunes. I rip and tag on a different machine and then import the product to my main iTunes. (My reasons are very simple... I often have family members using the machine) I do those chores on my MBP and transfer them across using a Firewire disk, in batches. I don't know if that has any bearing on my iTunes' quite acceptable performance.

Regards

TD

Once again Terence...extremely helpful info. I'm glad your not letting people's frustrations with the facts get percieved as "killing the messenger".

Let me try to sum something up here and let me know if I'm understanding it all correctly (some of this may be kind of rudimentary, but I figure I'd go through it ALL so it's in one place):

- Since we can't affect the software, people are going to want to optimize their hardware.

- For most this is going to happen through HD's since the only other option is buying a newer faster computer. Upping your RAM is good, but in my experience I went from 1.5 GB to 3.5 GB and didn't notice any performance gains in iTunes.

- Since the writing of the "iTunes Library" file and the XML file both happen on the startup disk, then you could see improvements (however nominal) by using a fast HD with a large buffer (like the Raptor). (Another poster reports that "I have found that there is SOME improvement with faster hardware ....Currently, I am using a 10K Raptor drive as my start-up drive, containing the iTunes program...") http://discussions.apple.com/message.jspa?messageID=3957832#3957832

- As far as the optimal HD situation for storing the actual iTunes Library I'm not quite as clear on the solution, but I'll give it a try and please correct me where needed. It would seem that you're fastest choices (in order) would be:

an SATA II
an SATA or eSATA HD
Firewire 800 external
Firewire 400 external
USB 2.0 external

- The faster these HD's are (spindle speed or RPM's/buffer size/seek times), obviously, the better chance you have at performance gains (however nominal). Getting anything above the common 7200 RPM HD is going to cost you though.

- A couple of other things to sum up here would be:

keep about 10-15% of your HD empty for better performance
delete unnecessary playlist, especially Smart Playlists

* Here's my question: If someone wanted to set up for a 400 GB library (knowing and accepting all the iTunes software performance issues) would there be any difference between buying one good 400 GB HD and buying, let's say, 4 100 GB HD's and doing a hardware RAID array?

This is a big question for me since it means either popping in one new HD inside my G5 or buying a whole eSATA setup (eSATA card & enclosure).


_______________________________________________________________________

* BTW, this whole discussion has helped me solve another problem with iTunes, DJ'ing and portability. Perhaps a bit off topic, but I thought I'd mention it here since so many DJ's have large libraries...

I have always kept my library on an SATA internal drive to maximize speed performance at my desktop. When I DJ or take my music on the road, I like it to be an extension of my desktop computer...identical to what I worked on the last time I was at my desktop. (For instance, maybe I made a playlist the night before for the event or party or rated an album of new music.) Before figuring out the whole eSATA thing, there wasn't an easy solution except to have a mirrored firewire drive...something that required a little upkeep. Now that you can pop an eSATA card in the right laptop, I think things might get easier.

Point is, after confirming that iTunes has issues handling large libraries and learning what I have in these discussions the past few days, I'm now planning on this strategy:

- keeping my iTunes library on an eSATA

- when I take my music on the road, I'll take that same eSATA HD with me and make a copy of my "iTunes library" file and the XML file (as well as the Album Artwork folder) from the startup disk and place it on the eSATA.

- when I get to where I'm going and setup my laptop and hookup the eSATA HD, I'll do the following:

before I start iTunes, trash whatever iTunes Library file/XML file/Album Artwork folder that is on the laptop and replace them with the current ones from the eSATA.
launch iTunes and I'll have the identical library that was just on my laptop on a fast drive. ( *you'll have to initially change your Library Location in iTunes preferences for this to work properly, but once you do it should be seemless)
if you don't do anything other than play music or make playlists, then you can just go home and plug the HD back in to your desktop. BUT, if you want to go back to your desktop with any changes you made to metadata, ID3 tags, etc. while you have been working off your laptop, then you'll have to go through the same process with trashing and replacing those essential files on your desktop computer.
backing things up is essential here. If you or your HD screw up anywhere along the way, you can always go back to where you were when you unplugged the eSATA drive.

I have been using this basic strategy (with copying and replacing the files) for some time now, and it works well. Now I'm just trying to figure out the best hardware to get to put it all together. If someone doesn't think this is going to work PLEASE let me know b/c I'm about to plunk down some cash to make it all happen.

I appreciate everyone's input...it has really helped. Perhaps this last part went too far off topic and should be it's own thread, maybe for DJ's.

This problem will not be rectified in Leopard, it's
not an OS problem. Give as much F
eedback as you can to Apple, maybe they'll
redesign it for iTunes 8.

Regards

TD

Thank you again Terence. You are the only one in the thread with so many posts, so that is why we are jumping on you, but your input is very much appreciated 🙂

And...you are so right. Apple Feedback is the best idea 😉

iTunes Feedback Page

I just hope they listen since iTunes is one of their core products.

Best,
IE



G5 2x2 / Pismo G3 400 / iBook G4 933 Mac OS X (10.4.8)

I wanted to add on to Echo's response here in giving thanks to Terence. Lots of education there and his reponses have been well-thought out and considered and educational for me as well.
Thanks Terence and the other posters here!

Best,
W. David

I like having a playlist for each CD I put into it. I have read and been told, that this is now a no-no :/
It's a "no-no" because it is simply a waste of time and will add needlessly to the database.
All albums already have their own "playlist" created by iTunes (and the iPod).

I like having a playlist for each CD I put into
it. I have read and been told, that this is now a
no-no :/
It's a "no-no" because it is simply a waste of time
and will add needlessly to the database.
All albums already have their own "playlist" created
by iTunes (and the iPod).

One more time...

It worked fine in iTunes 6. It is broken in 7...

It's slow or it doesn't work?

Why do you care if it worked in any iTunes?
It is pretty much pointless to create playlists for each album.

My guess is one reason it's much slower now because iTunes also looks for the album art in a different folder.

Keep in mind that iTunes is designed solely as a personal app and not designed for professional DJ use.

Why do you care if it worked in any iTunes?
It is pretty much pointless to create playlists for
each album.

Well, since you asked...

I like to look down the sidebar of my iTunes and see all the CD's I have in there. If I feel like playing one because it catches my eye...I do. Simple enough, no?

The other way entails me going to the library and looking through all the songs, or knowing exactly what I want to play and hitting search.

And as to why I care if it worked in any iTunes, it is just that, because it did work and now it does not work. 😉

G5 2x2 / Pismo G3 400 / iBook G4 933 Mac OS X (10.4.8)

I like to look down the sidebar of my iTunes and see all the CD's I have in there. If I feel like playing one because it catches my eye...I do. Simple enough, no?
Not as simple as viewing all the albums and artists with the browser. Go to menu View -> show browser.
Or click on this icon -> at the bottom right of iTunes.

This way you can keep your (non album/non artist) playlists in view on the side and see all your Genre, Artists and Albums up top.
Click any of those and you will see only those songs.
iTunes already creates a list of albums and artists for you to view very easily. Why recreate something for the sole purpose of slowing everything down?

The other way entails me going to the library and looking through all the songs, or knowing exactly what I want to play and hitting search
No, it doesn't See above.

I like having a playlist for each CD I put into it.

I can't find if this has been mentioned, but have you tried to get used to the BROWSE feature...or the "Grouped With Art" or the "Cover Flow" features?

Not as simple as viewing all the albums and artists
with the browser. Go to menu View -> show browser.
Or click on this icon -> at the bottom right of
iTunes.

This way you can keep your (non album/non artist)
playlists in view on the side and see all your Genre,
Artists and Albums up top.
Click any of those and you will see only those
songs.

I have seen the browser view before, but there are many songs that are split off by themselves, or with just a few tracks. So this view is good for some but not all of my music.

iTunes already creates a list of albums and artists
for you to view very easily. Why recreate something
for the sole purpose of slowing everything down?

I am not recreating something "for the sole purpose of slowing everything down?", that would be silly no? This is the way I have used iTunes for as long as it has been out.

Chris, I appreciate your trying to help, but once again, it worked this way in iTunes 6 perfectly. Something happened in iTunes 7 to break it. I do not think it is too much to ask that it works again in 7.

I have seen the browser view before, but there are songs that are split off by themselves. So this view is good for some but not all of my music
Not sure what you mean. If the songs are part of an album or an artist, they will show up when you click the album (or artist), exactly as they would in a playlist.

I am not recreating something "for the sole purpose of slowing everything down?", that would be silly no?
Yes, it would be silly but that is exactly what you are doing.

Many people start using iTunes after they are used to a different program. They try to do things the way they are used to, when many times, there is a far easier way to do it.
I'm simply trying to show you ways to take advantage of iTunes and use the features it does have, to make it easier.

I do not think it is too much to ask that it works again in 7.
Personally, I don't think Apple should be putting resources towards duplicating something that iTunes already does.
Everyone worries about "bloatware" and IMHO, this would be a perfect example.

However, if you want to create playlists for every album though, have at it. As you have noticed, it slows down iTunes.

You can send your suggestions to Apple here -> http://www.apple.com/feedback/itunesapp.html

Chris, I appreciate your trying to help, but once again, it worked this way in iTunes 6 perfectly. Something happened in iTunes 7 to break it. I do not think it is too much to ask that it works again in 7.

I think iTunes 7 changed the way cover art was stored. Also, the find album cover art feature added a lot of data (in a seperate folder now) to many people's libraries.

And are smart playlists new to iT 7? FWIW, I don't know if deleting all of your playlists is going to improve your speed. I deleted about 100 unnecessary playlists the other day (out of 200 total) without noticing any improvement. Now SMART playlists with live updating are another story.

And how much has your library grown since iT6? Just curious b/c I don't remember a distinct slowdown from iT 6 to version 7...for me I think it was more the amount of music I was slowly adding, and the amount of information associated with each file (as I continuously clean up more ID3 tags and metadata). I can't remember a particular time that iTunes started to slow down for me...it has been the proverbial straws on the back (except it hasn't broke yet).