Out Of Control Temp Files In Itunes Folder

I read somewhere that antivirus software could be causing this.

Set your anitvirus stuff to not scan the iTunes folder and see if it stops.

Katrina S. nailed it - it's related to your Anti-Virus software. I have 10,000 songs+ in my iTunes 6.0.1 library and never experienced any issues until I switched my anti-virus from McAfee v.10.0 to NOD32 v.2.5. I started investigating this after I saw CPU usage pegged at 100% whenever I quickly switched playing songs in iTunes, and also after I exited iTunes. It would take 60-90 seconds on my P4 2.26 Ghz with 1Gb of memory for iTunes to finish whatever it was doing and exit from memory. The CPU usage was pegged at 100% the entire time and the machine was virtually non-responsive.

While investigating this behavior, I opened the iTunes Music folder to see how big my .ITL and .XML's had become. the XML file was 14 Mb, and I had a bunch of files called "Temp File 1", "Temp File 2", etc. Each one was the same size as the XML file and were created each time I switched to play a new song in iTunes. When I exited iTunes, a new Temp file was created, too.

I just unistalled NOD32, and now the CPU usage stays at 5-10% max in iTunes, and iTunes now only takes 6-7 seconds to rewrite the .XML file upon exit. No more multiple temp files either. I'm going to submit a trouble ticket with ESET to get them to look at how NOD32 is interacting with iTunes. Not a good interaction at all. I've seen other postings related to high CPU usage while in iTunes, and this may be the same issue. Good luck.
Mike

Dell Dimension 2400 Windows XP Pro B&W Mac G3 with G4 upgrade

I was able to configure the AMON module in NOD32 to exclude the iTunes Music folder from virus scanning. All my Temp file and 100% CPU usage issues have gone away. iTunes is now snappy even with 10,000+ songs and exiting completes the rewrited of the .XML in 6-8 seconds (vs. 60-90 seconds with virus scanning turned on). I hope this helps other users.

Glad to help!

Don't know what NOD32 or ESET is, though. Same principle for any virus scanning software, though.

[edit: and I directed someone with the 100% CPU problem over here]

Message was edited by: me, myself and I

NOD32 is the anti-virus package that Leo Laporte formerly of Tech TV, now of TWiT podcast, Security Now podcast, KFI 640 podcast, etc. fame recommends to his Windows listeners.

Leo is a HUGE Apple fan and recommends that most folks just buy and use a Mac. For those of us who are gluttons for punishment, he recommends Eset's NOD32 for Anti-Virus protection. He's not much of a fan of McAfee, Symantec, etc.

This is the only issue I've found with NOD32 so far, but it was a big one! I'm glad the workaround is easy.
Mike

I think I figured out the real problem. I know some of you relate it to antivirus, high CPU usage, and so on. This is both correct and not. Let me explain.

But first, let me give you some specs of my machine, both hardware and sfotware, just to clarify some things.

Hardware
AMD Athlon XP 2800+
ASUS A7N8X Deluxe Motherboard
2GB PC3200 (DDR400) OCZ High-Performance Memory
830GB HD (80GB OS + 250GB x 3 data) + 500GB LaCie backup
GeForce4 Ti4600 128MB
Dual 20.1" Dell 2005FPW monitors
Logitech MX5000 Laser Desktop

Software
Microsoft Windows XP Pro with Service Pack 2 w/ all current updates
Trillian Pro 3.1 Build 121
Google Desktop 2
Logitech SetPoint
Virtual DAEMON Manager 3.47
Firefox 1.0.7

iTunes Setup
iTunes 6.0.1.3
32000+ songs (206+ GB)
music stored on F:\music (250GB SATA drive)
iTunes library stored on D:\My Documents\iTunes (250GB SATA drive)
iTunes Library.itl: 31.1MB (31,859KB or 32,623,357 bytes)
iTunes Music Library.xml: 41.8MB (42,852KB or 43,879,566 bytes)
iTunes.exe: 141,632K mem usage (129,188K VM Size) - approximate as it fluctuates.

The interesting thing that I found out is that it has nothing to do with the iTunes version. I've kept all legacy versions of iTunes since 4.6 and all previous versions cause the same problem. What I did find though is that my system will constantly write temp files at a rate of about 2-3 a minute depending on what I'm doing with iTunes.

Action: Play a song.
Reaction: temp file is generated, then disappears.

Action: Change the rating of a song.
Reaction: CPU spikes. Two temp files are generated and do not disappear.
Addendum: Each temp file is the size of the iTML xml file.
Reaction 2: CPU spikes. One temp file is generated and disappears.

Action: iTunes moves to the next song.
Reaction: CPU spikes. Temp file is generated and disappears. Song may stutter.
Reaction 2: CPU spikes. Temp file is generated and does not disappear.
Addendum: File is the size of the xml file.

Action: Change rating of a song and change the playing song.
Reaction: Temp file is generated and disappers.

Action: Skip through a song.
Reaction: Low CPU spike. Otherwise, no reaction.

Action: Create new smart playlist.
Reaction: Temp file is generated and stays.
Addendum: File is size of the xml file.

Action: Delete smart playlist.
Reaction: Two temp files are generated.
Addendum: One is the size of the itl file, the other is the size of the xml file.

This leads to several observations:
1. mp3 playback is a low CPU process of iTunes. This seems fairly logical as it is the most crucial factor and users will want to do other things with their computers besides only listening to music.
2. Any change to song information in the interface will cause a spike in the CPU and a read/write of both the itl file and the xml file.

I'm curious what kind of Windows systems they test their iTunes software on. They must be high-end machines to not catch issues like this.

This leads to several (linked) conclusions:
1. It seems as if this only happens to users with large libraries. Since the CPU is caught in an intensive process, it seems like the software is attempting to write from memory to disk. This makes sense (sort of) as the intention is most likely an attempt to preserve any immediate database changes if the software or system crashes.
2. The system attempts to write both the itl and xml file at the same time. Why it does that is beyond me - the xml file is merely an industry standard format of the exact same data as the propriety itl file. It also ***** as an xml format (if you've ever taken a look), but that's another issue.
3. The file generation depends on what stage the system is on when it stalls. If it's trying to write both the itl and xml file, then both will be generated.
4. Files are not cleaned up when the CPU spikes, resulting in a delayed write of the temp file(s). This seems to suggest that the "cleanup temp file function" is set on some sort of fixed sleep cycle and is executed after a certain amount of time the "save to itl/xml files function" is executed. The more operations iTunes has to perform, the more time it spends writing the file(s).
5. Windows locks files upon read/write. If you've ever tried deleting a file that's in use (such as an open document), you know what I'm talking about. A perfect example of why Microsoft creates all those .tmp files in the same directory as the document you're editing: the OS has problems reading and writing to a file at the same time, so the temp files are all consolidated after the file is closed.
6. I'm suspecting that this behavior has some correlation to the way Windows deals with files. If you look at Windows programs such as bittorrent, that break files into multiple chunks and performs simultaneous read/write commands to the same chunk, then you will not notice this behavior.

Here's what I think happened with iTunes development. The developers realized that the OS couldn't handle simultaneous read/writes (or something like that) and decided to use a temp file system. They wrote the software to create a temp file to write changes, overwrite the existing itl or xml file with the temp file, then attempt to delete the file. For some reason, the delete portion of the code does two things:

1. It is set to delete at a certain time, and if it cannot, then it just ignores and does not retry.
2. It ONLY attempts to delete the file that was most recently created and does not try to delete any other temp files in the directory. While this approach makes sense if other programs are writing temp files to the same directory, it still seems kind of silly.

There doesn't seem to be any possible user solution. Maybe I (or someone else) may attempt to write some sort of background script (a plugin of some sort?) that will run after certain periods of time and ensure that all temp files are deleted from the directory. It might be a bit tricky. The other solution is that the iTunes development team fixes their temp file deletion code.

For those of you using antivirus, my explanation makes sense even if you only have a small library: the antivirus software is attempting to check the file for viruses (performing a read), thus locking the file while iTunes is trying to delete it. Usage may vary, depending on your antivirus software, as some will check only upon a file write, while some will check only upon a file read, and some will check for both reads and writes.

Hmm... maybe a better solution is that Microsoft writes a better OS. Oh, wait.. this is the best they got so far. Maybe Vista will be better..

The simple solution? Get a Mac =) I support Apple and have a PowerBook, but still use a Windows desktop for various reasons.

PowerBook G4 12 Mac OS X (10.4)

Hmm... Weird. The forum won't let me edit my old post. Seems buggy.

I forgot to mention one other thing that lead to my theory and its conclusions. I've been getting this error where it says something along the lines of "The iTunes Music Library.xml could not be written. You do not have permission to make changes to the file." I don't remember the exact wording, and I don't have any new files to overload the system and get it to pop up. But it seems like it occurs when there are several writes in queue with reads. I think iTunes tries to read/verify after every write.

Michael: You're essentially correct, although I'll summarize anyway. When iTunes has to make a database change, it rewrites the entire file out as a temp file, then does renames and such. If some other program locks this temp file after iTunes releases it, then iTunes is unable to delete it and gives up, leaving it there.

Warning: Heavy programmer speak lies ahead. 🙂

First off, it's really not the fault of the way Microsoft OS's do file handling. ALL current operating systems, including Mac's, do file handling and locking in much the same way. There's just not enough difference between systems on that level to be able to say that it's Microsoft's fault here.

This is basically an iTunes problem, although I would hesitate to call it a bug. This method of writing to a temp file and then doing file copy operations is, to be blunt, an extremely stupid way of doing things. I can understand why it's done this way, though, as I've seen code that works in much the same way. I blame the Object Oriented Programming paradigm in the long run. The notion of seperating everything out into objects with simple operations leads programmers into this trap eventually, and breaking out of it is a bit tricky. Their database object probably has a flush or write function which does this. It doesn't analyse what change was made to the database to determine the best way to write the file, it simply writes the file in a way that will always work. It probably even throws an exception when the temp file cannot be deleted, because any good object would. However, that exception is not a critical one and so it gets ignored by objects which call that database write operation.

In other words, it's not so much a bug as it is a design flaw. These sorts of flaws are commonplace in OO programs written by programmers who learned OOP to begin with and have never done any serious non-OOP work. It's not anybody's fault, it's simply that the methodology leads one to not consider these weird edge cases so much. OO programmers tend not to optimize for this sort of condition, because each function is so simple, so basic, that it's hard to see, from just looking at the code, where optimizations would have real benefits.

But that's just me waxing philosophical now. 🙂