Podcast backing track volume spikes.

My podcast backing track spikes in volume whenever someone isn't talking. I have tried using ducking but it persists. Any suggestions? The tracks are of random time periods so I have already gone in and have to raise or lower volumes depending on quality.