Continuity mouse reverts to Mac mini with MacBook keyboard

Continuity mouse/keyboard issue

I remember this working ok in the past, but if I drag my mouse over to my second computer (MacBook) as soon as I start using the (inbuilt) keyboard for that machine that the mouse is now controlling, the mouse zips back the primary machine again! (Mac mini). Is there a setting im missing somewhere? it seems like a strange behaviour.

thanks for any help you can give

Nick