Audible click when playing sounds

For a few months now, I've had an FiiO Olympus E-10 USB headphone amp. The reason for getting it was convenience; it allows me to just unplug the Thunderbolt cable from my MacBook Air and leave my headphones hooked up to the Thunderbolt Display through the E-10. The audio quality is actually quite good, but for some reason I get an audible click every time audio stops.

• When a system sound has finished playing, there is a click
• When I pause a YouTube video or when it ends, there is a click
• When I play music from iTunes and a track pauses or ends, there is a click.

It's really quite annoying.

Additional data points :

• When I have the headphones connected to my audio jack on the MBA, the click does not happen.
• When I have another pair of headphones connected to the E-10, the click also happens.
• No specific drivers were installed to support the device. None are available on the manufacturers site either.

In my view, all of the above appears to point towards a hardware issue.

Any similar experience?

Any thoughts on how to resolve it without buying a new device?