Audio Issues USB

This has happened to me a few times with input and output using the USB ports for headphones and Mic. Basically we start to sound like a robot. In this case I was using a Zoom H4n as the mic connected to USB. I am also running an audio program and when the issue happens a restart of that software will stop the issue BUT there are other times where I am using USB headphones and the audio I am listening to sounds like that - only fix then is to switch the USB port. Any thoughts on what could be causing this? It does happen on both USB ports.