Yeah, this is frustrating for me, too. I tried a few different variants: keeping everything in Aperture, keeping only videos in iPhoto, keeping videos in Movies folder, etc. Reason being I'd like to be able to sync some videos back to the iPhone will keeping access available for iMovie - which is tougher than it should be. Now that the iPhone does HD videos, Apple needs to either adjust iPhoto to work with videos better or work with iMovie to actually function as a movie database as well as editor.
For now, I, too, have my stuff strewn about on my Mac but I'm trying to consolidate as much as I can into iMovie events on an external drive. I have a bunch of video clips of my 2 year old and I've decided to just group them by year and then put together an edit of each to then put into iTunes so I can view them on AppleTV as well as sync to my iPhone. I figure I won't ever sit down and watch every single clip so I might as well put together the "highlights".