MIDI Controller Latency after pressing Record

I'm having an issue with latency on the notes from the midi controller on a specific project. Whenever I restart the project, or change the audio settings it's back to normal, but as soon as I hit record, the latency kicks in. This applies to both third party and native Logic patches and it continues to playback the note on delay even after exiting Record mode. The only way to get it back is to change audio settings. Audio buffer of 512 and 256 were producing similar delay lengths, while a buffer of 32 seemed to shorten it a bit, but it's still very noticeable when trying to lay down a track through midi against a metronome (slightly under a 16th note at 142 bpm). I don't really get this on other projects. I have one track on this project in Binaural Panner, but that's the only major difference I can think of between this session and others where I don't experience this issue. Could this be why this is happening?