I've noticed the same problem on my Mac Book Pro. When I clicked Videos and then clicked my Movies directory, Front Row would hang for a few minutes with the spinning wheel. It would also often hang with the spinning wheel when I clicked other items.
I was later playing with "Glow Worm FW Lite", a free application that can alert the user to any new network connections and block applications from making connections. I fired up Front Row, and noticed that sometimes when I clicked on various items in Front Row that I did not expect to make a network connection, such as when I clicked the 'Movies' link under Videos, Front Row would attempt to make connections to apple.com servers.
When I set Glow Worm FW Lite to block all network connections emanating from Front Row, this suddenly fixed all the problems I had with Front Row hanging when I clicked 'Movies' or browsed through a few folders of movies.
I think the problem is that, for whatever reason, Front Row attempts a connection to Apple's servers (several connections to different servers over the course of a Front Row session --I should have kept the logs and packets to determine why it's phoning home). My network connection is iffy, and Front Row sits there for minutes waiting for its connection to timeout. When Front Row is unable to make the network connection, it gives up instantly and allows the user to continue using the program without delay, and apparently without adverse effect.
You can get Glow Worm FW Lite here:
http://www.apple.com/downloads/macosx/networking_security/glowwormfwlite.html
Try setting it to deny all network connections from Front Row and see if that doesn't fix your problem too. The way to set up that rule in the firewall is to first run Glow Worm, run Front Row, exit Front Row when Glow Worm pops up an alert, respond to the alert, then switch to the Glow Worm application, and choose "Plugins > Rule Editor > Rules..." from the menubar. Double-click the rule that was just established for Front Row, change the Options tabs setting to "Deny" and delete any entries in the "Connection" tab (which will set it to block all connections).