Constantly hot audio input issue in Logic

Whenever I don't have my (USB) audio interface connected and I just plug headphones directly into my MBP and fire up Logic to do some mixing, the built in mic on my macbook is hot and extremely sensitive, despite having no tracks with input monitoring turned on.

I always have to go into my system prefs and turn input sensitivity all the way down to remedy this (and then have to remember to turn it back up when I'm on a work call the next day).

Anyone else run into anything like this and have any suggestions? Tried searching but since it's a bit of an odd topic couldn't find anything.

Thanks!