Why are My iTunes Downloads so Slow?

I decided to download one recent Doctor Who episode in HD to see what it was like.

My current internet is ADSL2+, and I live practically next to the local exchange. (There's one building in between.)

Whenever I download something in Steam, it generally sits around 1.2-2.0 MB/s.

At that rate, I should be able to download an episode in 20 minutes.

But no, it took 3 hours. Or that's what it said when it started, and for the first hour and a half. I went to bed after that and it was finished in the morning.

@Endurion - This was exactly my issue too. Google DNS does slow down iTunes downloads dramatically. I was able to reproduce the issue with the same results as you. I'm not longer using Google DNS.

So what? It still does NOT apply to everybody as DNS lookups are not done for every packet. When a site have been looked up the result will be cached and the DNS will not be consulted again very frequently.

I don't use Google DNS and I have extremely slow speeds, taking 2-3 hours for a rented move to download.

This is just want you want to belive is happening, not what is really happening. Especially so in a fashion that others can learn from.

So how many of you guys reported this as a bug? Apple listen to compliaints if done informatively and with detail. If you're not happy with the services tell Apple! I did.

And, on reflection, DNS could impact download speeds - if it's an interaction between Akami and Apple. There isn't a single name/ip mapping for Apple's servers. Based on your particular geographic location, Akami (and similar services) will change the ip returned to route you to the most optimal server. Local DNS will work in this case.

But if you use a DNS provider in a remote location, that breaks the geographic routing model. By using Google DNS, you're probably triggering a route to a server located out in California, near Google's HQ's - which could be a very slow route.

It looks like Level3's public DNS servers, 209.244.0.3 and 209.244.0.4 route you to a DNS server physically located near you. I can't be sure because I actually live near where those IP's terminate, but after switching to them, it looks like I've got decent download speeds again. Try those, or try finding local DNS providers and use those servers.

I've heard some ISP's don't use local DNS servers either, so that could also be causing the problem.

Oh yeah, but what if you don't use Google DNS in the first place? Also, the nearest server should be used no matter which DNS is used.

Also, Apple are no newbies when it comes to building high performance solutions on the web. We, the customers of Apple, are in our full right to expect a very nice service level, better than many others. If Apple want to ruin the brand that's their right, but I will complain whenever they try to pretend their service should be expected to be sub par.

All of you that takes sh** from Apple are responsible for the service we all get. Complain! In a nice way, but make your point.

I haven't fully validated the theory yet, but after talking with some engineers that used to work at Akami, it seems like this may be at least part of what's happening. My own experience buildling networks is that it's never just one thing - they are complex beasts, and can respond in unexpected ways.

If the DNS server is in (say) Boston, and you live in Texas, Akami's nameservers will return a route for a Boston end point (in other words itunesdownloads.apple.com will have different IP's based on where you are in the country). So if your ISP, for example, has centralized DNS servers located in a different location, the same thing will happen.

And that could also explain why we see such variance over time - akami and other folks like them, do dynamic network load management - you can get different IP's at different times. Add in router and OS DNS caching, and trying to figure out exactly where things are going at any given time can be a challenge.

To test the theory completely, we'd need to have a network monitor, check the IP location where the downloads are actually taking place when it's slow. Then switch DNS providers, reboot your router and your computer, and continue the download. It may very well change.

I do agree though, that since this is a widespread issue, Apple needs to diagnose the issue and then drop a KB article - it may simply be 'don't use XXX DNS services', but yep, their user experience is their responsiblity (even if they don't own the network).

You are right on the money lhotka, the only other variable I will point out to make sure people understand -- the reason for the slow speeds is most likely (I'd say 90% of the time) because the Akamai server you are being directed to is overwhelmed NOT because when you got sent to one far away. The change in DNS just got you to go to a different Akamai server set - thus the variability of everyone's results. Even if you switch to Google, google has some 20+ sites they serve from via Anycast -- so it will work for some of you but not all of you.

I agree though, that this is Apple's issue to solve and they've dropped the ball. If Apple is reading this, I do suggest they go look at a company named Cedexis (www.cedexis.com) who sells services to companies like Apple to solve this very problem. (No I don't work for them but I do have experience with them). They have the ability to detect this type of issue.

Ihotka, There's something wrong with your reasoning. The web server normally have no knowledge of which DNS server the iTunes client machine asked to do the DNS lookup. Therefore the route to the client can not be based on the physical existence of or the network spatial connections to the DNS server.

Rather the route would be based on the network location of the client — as far as that can be known — and different algorhitms for determing the best server to do the serving in question to that network point. Which could be both in the web network and a contracted outside party like Akamai.

I suspect that all this is only part of the issue, and that there may very well be bandwidth issues at Apple's sites (which is why Friday and Saturday nights are always slowest).

Let me try to explain how the Akami type of architecture works. Remember - the iTunes client 'pulls' information, the server doesn't push it (the iTunes servers don't have a clue where a client is located).

For right now, let's ignore caching.

When iTunes on your machine wants to download something, it triggers a DNS lookup for the remote server. Let's say that the name is something like 'itunesdownload.apple.com'. That query goes to whatever DNS server you have selected - for now, let's say that's a local server from your ISP. That DNS server is fed information about what IP address to return to that server name from the next DNS server up the chain, until a server is found with an IP address mapped to that particular name.

In this case, Akami or one of their bretheren sits between the local DNS servers and the authoritative apple.com nameserver. That Akami server will return an IP address based on the then-best route between the requesting DNS client and one of the several duplicate itunes content servers scattered around the country. The 'best route' is determined by Akami's algorithms using multiple criteria, one of which is the location of the DNS server making the request - location is both roughly physcial (IP addresses don't port around the country like phone numbers do), and topological (network location). That determines the latency to the server, which (combined with dynamic monitoring of server load) is the secret sauce that makes video downloads and streaming video work. All of this is dynamic and can (and does) change on a regular basis. Any of the big content streaming sites (ESPN, YouTube, etc) all work the same way. itunesdownload.apple.com does not have a single IP address - it has several (all at the same time), but the one that appears to your machine is dependent on the particular chain of DNS lookups triggered from your client, which changes over time.

Caching introduces delays into how often the name/ip mapping is updated. That's why rebooting your computer (or manually clearing the DNS cache) can help improve speeds - but only if your router isn't also caching DNS lookups (Airport's can do that, for example). So in periods when the network and server loads are changing rapidly (i.e. friday night when everyone's downloading a movie at the same time), you may be hitting a less than optimal IP address.

So when someone uses a remote DNS server - let's say a Boston Comcast user decides to put in a PacBell DNS server, when the Pacbell DNS server is queried, it will return an IP address that is best for a west coast PacBell user. The DNS lookup is based on the DNS server making the request, not the client. Thus the client in Boston will route across the country, to a sub-optimal web/download server (which has no clue where the client is located).

Can this be fixed? Maybe - Apple could drop the Akami dynamic routing and do application based network load balancing (i.e. iTunes would have to ping each of the various download servers in order to figure out which one to connect to). That would bypass the caching problem, but introduce other issues.

In any case, it's definitely an issue, and Apple does have a UX problem that they need to solve.

And just one more note: I just turned on Little Snitch. iTunes appears to be downloading from www.ign.com. nslookup gives:

Address: 204.0.87.138

Address: 204.0.87.139

So at least part of this is definitely tied up in Akamai (I've been spelling it wrong - oops) - DNS can absolutely have an impact, both the server chosen, and the caching timeouts.

I appreciate all the explanations, proposed solutions and technical nuances of solving such a large technical problem.

My response? So what?

This is something Apple needs to either 1) fix or 2) make completely clear at the time of purchase* To expect that a download from the iTunes store would proceed at a pace congruent with files of similar size from other sites is entirely reasonable. That they make take, say 10x or more is reprehensible at best and bordering on theft at worst.

It shouldn't be up to the consumer to debate and tweak ways of solving this issue. Make the stuff available as torrents if they can't or aren't willing to cough it up for bandwidth required.

It's not my problem, don't ask your customers to solve it or endure it. Just FIX IT.

* suggested verbiage (CC with attribution): "Downloads from the iTunes Store will be mind-crushingly slow. If you are leaving on a flight within the next 16 hours, we recommend starting your download now and purchasing a DVD of the movie of interest at the airport. You're download will be waiting for you upon return."

I have no connection in the logs to ign — I also peruse LIttle Snitch — but rather to a1442.v.phobos.apple.com. I think your theory is long overplayed if ever true. A company the size of Apple can't afford not to use application load balancing nor can Akamai when they're involved. It involves too much data.

In addition you seem to be misinformed on just how application load balancing in cluster server setups is done these days. Pinging need not be involved.

It took me just short of 3 hours to download a 2 hour feature in HD last night (rented). That was about 1 hour too much with my connection (8mbit going in).

Andy2k wrote:

DNS Servers have NOTHING to do with your download speeds.

DNS does have an impact on download speeds when you are downloading from a content delivery network e.g. Akamai, Level 3, Edgecast or Limelight.

You are right that DNS is like a phonebook. But Akamai uses this phonebook every 20 seconds, to optimize content delivery from the nearest topolocigal location, based on network status, congestion, etc.

Read and be wise: http://www.akamai.com/dl/technical_publications/network_overview_osr.pdf

I have read through the whole thread now, and it seems that a lot of people have misconceptions about DNS. DNS was not made to have an impact on download speeds [1] - BUT - several CDNs are using DNS in a way that makes content delivery dependent on DNS [1,2].

If you are experiencing slow downloads from Apple, it is likely because of external effects like change in standard DNS, or network congestion.

Are you "clever" enough to change your DNS, you should be clever enough to understand the impact DNS has on download speeds from different CDNs.

If you are experiencing slow download speeds, my first question will therefore be: what kind of DNS server do you use? My tip: use your local ISPs DNS.

If you want to know more about how content is delivered in the internet today, read [3].

[1] http://queue.acm.org/detail.cfm?id=1647302

[2] http://www.cdnplanet.com/blog/which-cdns-support-edns-client-subnet/

[3] http://www.akamai.com/dl/technical_publications/network_overview_osr.pdf

I have a 50x5 Mbit cable business class modem, from Time Warner, in my house (doing that give me priority bandwidth over other residences).... Also as a business class there's no data caps or port blockings. The most I EVER get from the iTunes Store is 4-5MBytes/sec (about 35 Mbits)... Usually I get 1-2 MBytes/sec to iTunes or any device

My guesses are

1) Apple just doesn't have the bandwidth during certain times

2) I notice they throttle accounts. If I download an entire hd season, I start out fast, then it slows after a few hrs... Im guessing apple has a max data cap per day then slows you down.

3) I also don't think Apple is capable of handling all the new 35-50Mbit connections that are now readily available from the big cable companies.

Im in Charlotte, NC, USA