From what I understand, this entirely has to do with a few factors:
1) The time you are trying to download items. Peak times mean more people, which means the servers have a bunch of people accessing them.
2) Your location. You could either be close to an accessible server for data, or you could be far away. The further away to the server you are having to download from, or the bounces it takes to get you to it, the slower the download is going to be.
3) It really still could be something with your network or anti-virus/etc on your computer. Try, perhaps if you have a firewall or router, to ensure that it is not blocking or slowing down iTunes in any way.
This article may be useful:
http://support.apple.com/kb/HT3204
Some others that may help:
http://support.apple.com/kb/TS1379
http://support.apple.com/kb/TS1490