There are several ways to do this.
First go to iMovie/Preferences and make sure the Advanced Tools are enabled.
The simplest way would be to select the frames of video in the Event that contain your audio and drag to the project. Drop it on the approximate frame (in the project) where you want it to start. A pop-up menu will appear: choose Audio Only.