I have the same problem with my MacBook. The problem first presented itself around the time I started using a wireless network, right around the latest firmware update, and I'm guessing it's the network that's slowing things down.
The funny thing is, that it only affects the "videos" - section and nothing else, and there seems to be no correlation between the freeze-time and the amount of video files.
The other thing I noticed, is that if you press the remote commands in advance, ie. twice down and play after hitting "movies", there's no freezing at all. Still, trying to get out of the Front Row after watching a video can be tricky if not altogether impossible.
Front Row used to be the first thing I'd show to PC users, but not anymore.