why does my mouse freeze all the time?

I have a weird problem with my new 27" iMac late 2013. all USB mouses I try keep freezing randomly all the time while my magic trackpad keeps working (Of course I have tried to disconnect the trackpad and it still makes no difference). I get no console errors and I have tried all USB ports I have available, always the same issue. I have tried my cyborg rat 7, my razer mamba, my old pro mouse and some generic usb mouses, every time I get these freezes. Reboots don't help at all, the only thing that helps is reconnecting the mouse which is very annoying over time.. I had all drivers installed properly and still this happens every 5 minutes or so..