Good you figured out your modem issue. Fast download rentals in Apple TV are just around the corner for you.
For the 5 GHz band, it depends on which revision of the Airport EXtreme you have. But you should find it under Airport section in the Airport Utility. Once in that section, select the wireless tab.
Then, depending on the version you have, you will either find it directly on that page where you can select the type of network used (802.11 n/b/g compatible, 802.11 n 2.4 GHz, or 802,11 n 5 GHz)
If it's not there (because you have the simultaneous dual band model) , you will have, in the same page, a button named "wireless network option". Under that, you will be able to activate the 5 GHZ network and give it a different network name (ex : Airport 5GHz). Also, make sure the wide channel option is checked.
On either one or the other, make sure you reconfigure your wireless network in the Apple TV to make sure you are connected to it on the 5 GHz.
If the Apple TV is not that far away from the router, you might find the 5 GHz band faster and much more reliable than the 2.4 GHz one depending on devices you have that can interfere and how many people in your neighborhood run a wifi router. There are 11 channels in the 2.4GHz band. Most of your neighbors who have a wireless router, they are set to auto channels which means that they will be mostly on the channel 6-1-11. If there are too many people around with that, they can interfere with your network and slow down the speed or just get your connection lost. The 5 GHz band have 149 channels and today, very few people run under that band.
Decently fast internet connection and a fast reliable wifi network are the key to comfort with iTunes store HD rentals. This is considering that your ISP doesn't throttle speed from iTunes store servers.
Looking forward to see your results.