Great. Glad that the workaround did the trick.
Audio speed is measured in sample rate. A voice memo is recorded at 44.1 kHz (at least on the Mac), that is slower than an audio recorded at 48 kHz. So, with an audio editor app you can adjust the sample rate to vary the speed of the audio.
If you open the audio voice memo clip in QuickTimePlayer, and do a Window/Show Movie Inspector, you can see the sample rate and the data rate of the audio. Perhaps you can then export the video clip from iMovie using the custom Quality setting to match up the data rates. I've not tried that, but possibly it could work.
I have found that it is just easier to use the speed adjustment slider method that I suggested, as long as it renders a satisfactory result.
-- Rich