I had the same issue. I initially noticed it when I saw that mdworker was continuously spiking in CPU usage, then going back down. I used "sudo fs_usage -w -f file sys mdworker" to figure out that mdworker was indexing the second partition on my Mac. This partition contains another installation on OS X. Turning off indexing on that partition solved the problem. Upon booting into the other installation, I found the root of the problem. I had turned FireVault on, but I didn't allow enough time for it to finish encrypting. I allowed it to complete the process, then booted into my main partition. I reenabled indexing for the second partition, allowed about 10 minutes for the indexing to complete, and now the problem is gone.
In short, if you have a second partition with another installation of OS X with FireVault enabled, be sure that it actually finishes encrypting.
I realized that since this is happening with an external drive so we probably don't have the exact same issue, but see if it stops if you disable indexing on the time machine drive.
