Two Thunderbolt Displaysand USB Audio Issue

I have a brand spanking new Macbook Pro 15" Retina (2.7 Ghz, 16GB RAM, discrete graphics chip). I also have two thunderbolt displays, one is about a year and a half old, the other is pretty new.

I also use an Apogee Duet 2 USB sound card.

If I plug the USB soundcard into my Thunderbolt USB, after a few minutes the audio is all distorted. I've so far resorted to using a USB port on the MBP itself, which fixes the issue.

Is there a "Best practices" guide for how to connect up these displays? I currently have them daisy chained, so I have just the two cables going into my Mac which I wanted. Is there a better way to hook this up for reliability sake?