How to sync iPhone video with GarageBand audio

I recently recorded a video of myself playing/singing. As I played, I recorded a voice track and a guitar track in GarageBand. Then I used patches and mixing to enhance the tracks. Then I outputted an MP3 file.

My intention was to detach the video's audio track, then use the GB recording as my primary audio, but perhaps mix in some of the iPhone's audio to add some "color" of a live performance.

In iMovie I detached the video's audio track. Then I went about trying to sync the video with MP3 audio. After a lot of very painstaking work I was able to sync up a part of the video with my GB file. I used a distinct point (where the vocal comes in), and at that spot the synchronization was perfect, both sound-wise and visually.

However, as I play the video, the audio very gradually gets out of sync with the video. Somehow there is a slight time difference between the video and the GB audio.

Can anyone explain how this could be?

If it's just not going to work using an iPhone for the video, what do you recommend?

I want to be able to make a video, but have the audio part be editable. I am fairly good at using GB, so I want to stick with it. But obviously, I need the audio to sync perfectly. The audio would be the best recording I can produce in GB, but I'd like to be able to, in places, mix in some blend of the video's audio.

To summarize, the video's native audio track, even after being detached, syncs perfectly with the video's visual part. However, I cannot get the GarageBand MP3 file to remain synced (as it plays) with either the visual or audio part of the video. It gradually drifts more and more out of sync.

Thoughts, anyone?