I had exactly this problem, importing the video both from within camera, and taking the sd card out and putting it in a reader. I quit Imovie, reopened it. Then I reimported the video. SAME PROBLEM - no change. Then I restarted the computer, opened Imovie, reimported the video - now for the fourth time. AND EVERYTHING WAS FINE!! The audio was perfect.
So what caused this in the first place, and what happened in the reboot that fixed it? Beats me. One moral of the story is no matter how good Mac OS gets, rebooting the computer is always a good thing to try. Another moral of course is, don't delete any files until you know your imports are good and you have them backed up.