FWIW,
I had one channel unused on my Tascam US-122, and was using the other with a condenser mic with Phantom Power on, and I got a lot of white noise if the unused channel is turned up with nothing plugged in. Thinking it was defective, I got another US-122 and it does the same thing. So, now, when I use phantom power, I have to make sure the unused channel is turned down. Your problem "could" be a similar thing.