I found this in Mac forums this works
Originally Posted by sorsor
I know this is old, but I ran into this earlier and came across this thread in a google search. As it turns out, it takes forever because it converts the video to .mov container (DV format). I found through further searching that you can instead just drop the videos into your imovie project data folder and it will use them without converting. It still takes a few minutes to process (1 hr .m4v 1920x1080 clip took 12 minutes on my new Air, instead of 9 hours as previously estimated through normal import), I suppose indexing or something, but much better.
Specifically, in your home folder, go to Movies -> iMovie Events -> (project name), and drop in your video there. When you start iMovie it should start processing the files, but it didn't actually convert the files. They just popped up in my event library ready to use.
This has to be the stupidest thing I've come across, as the forever conversion seems completely unnecessary for the application to function. perhaps it's a safety net for certain other formats.