Q: KVM usage for wired Apple KB and Mouse with iMac late 2009 and Mac Mini i7 2012

I have bought an older iMac off a friend. I aim to use it in Target display mode. I own a hardwired Apple keyboard and mouse for the Mac Mini. My question is will a USB KVM work for enabling target display mode then switching to use on the Mac Mini. It is quite annoying having two keyboards and mice just to enable this function. I will connect the display via a minidisplayport cable I already have so I am just asking in terms of the keyboard and mouse. My worry is that when using the KVM the iMac will come out of target display mode when I switch.