Here are some answers:
1. I believe that Apple was well aware of a problem, but couldn't reproduce it with the appropriate instrumentation to debug it in their locations. Also, the randomness of this made it look like simple fixes, such as changing wireless channels, resolved the issue (but it turned out that in the long run it didn't). And, this impacted some, but not all customers. Also, this problem fell at the intersection between Netflix and Apple, which is never easy. The fix was for Netflix to no longer serve content from one of the three 3rd parties to ATV2s. Therefore, there isn't anything for Apple to do, and no update needed for us. In the long run, I don't know what the plan is for bringing this 3rd party back on line. If I were Apple, I would work on an error recovery/debug/reporting update for this type of problem in anticipation of future hiccups. The Netflix statement about the low ATV throughput threw me off for a while too, until the ATV2 development team explained that they used an adaptive, variable rate bit scheme, so this rate is low by design, and was/is not an indication of a problem, which now makes perfect sense to me. I hope the Netflix support reps get the word about this.
The CDN thing is actually simpler than it sounds. When you hit "play" on a Netflix movie, you don't get the movie content directly from Netflix. Netflix contracts with three different companies to host copies of the content, geographically distributed around the country. Your content comes from a server run by one of these companies... a server that has it and that is close (in a network sense) to you. The more popular the content, the more servers it is cached on. The issue of OpenDNS or Google DNS messing up your geographical location, instead of using your default ISP one, and the role it plays in Netflix assigning you your CDN server for the content, is a murky one. I have found a lot of voodoo surrounding this topic, but not alot of hard data. Personally, I think that Netflix and the CDNs are smart enough to use our IP address to get a rough geographical location, and can use some quick supplemental measurements if needed, to find the right CDN server for each of us, for most cases. So, I don't think that our DNS server choice, as long as it is reasonable, makes a big difference. But, it would be an interesting experiment to run (I might have a student work on this question this Fall).
2. I don't know, and I wish I did. My best, wild, guess is that the "problem" CDN vendor had a bad network component, like a router, that impacted some of the routes from it (like to you and me), but not other routes.
3. The ATV2 does heavy buffering when you play iTunes content. However, they chose a different path than Roku for Netflix content. Whether this was the right move, or not, is probably subject to debate. Personally, I think that the adaptive variable bit rate approach, like the ATV2 uses with Netflix, will be the right choice in the long run, and (totally speculating here), it might be an approach that, because it minimizes required bandwidth, allow Apple in the future to server more and more content from their own data centers in the future, without the need for these third-party CDN caches, allowing Apple to be more in control of their user's experience. I think we are seeing some growing pains with this new technology.
From my perspective, we should have two views of Apple here. We should be hard on Apple for allowing this problem to fester for so long. Given the 85,000 views on this thread, this was a big issue and must have cost them and Netflix a lot of support money, and cost their customers a lot of very frustrating wasted time. But, we should be also thankful that Apple stood up to take ownership of this issue even when, as the dust settled, it was totally (or at least mostly) Netflix and their CDN vendor's problem.
I forgot to mention the worst part of this experience: Apple asked me to watch a list of popular titles (because they would be cached in many places), so I had to sit through a bit of Kick-***, Dora The Explorer, Easy-A, and Curious George 2 🙂
