I've noticed this same issue. I don't watch a whole lot of YouTube videos so I only realized this just recently. Basically, when on wifi everything is fast, web browsing, mail, etc. except for YouTube. It takes 5-10 minutes to load a 1-2 minute video. This is the case with every video I've tried watching when on wifi.
My internet connection is not the issue, it is 6Mbps down, 1Mbps up and almost always tests out at the rated speed, if not it's very close. I don't have any issues with my router and am able to max it out to spec. when transferring files over my local network. This has to be an issue with the YouTube app specifically. Here are some details on my configuration:
iPhone 3GS, OS 3.1.3
Comcast internet 6 down, 1 up
Buffalo WHR-G125 router (G-only, WPA2 encryption)