Scarlett 2i2 and core audio battles

I have a 2012 Mac running 10.11 and have had this constant running battle between the audio interface and the core audio. What happens is that everything will be running smoothly then there will be a couple of clicks and a announcement comes up saying the "built in audio" will be used for this session, and it will default itself to built in audio. I have contacted Focusrite about it, they bascially said it wasn't a fault of the interface. I have also done the sudo kill command in Termonal, nuked core audio from the activity monitor, and sometimes it will work fine for a while then of course in the middle of a session, it will decide to default to built in audio.. I have gotten frustrated and just pulled out the interface cable, waited and plugged it back in again, which engages the interface, but sometimes, it will just bounce back and forth between trying to decide to use either the interface or the built in..

Right now it's sulking, so I can't finish my session, btw it does the same thing with my M Audio interface too, so it is a core audio issue which i'm not smart enough to figure out...