Siri won't work with podcasts playing

When I am listening to podcasts, I cannot invoke Siri from the headset. Sometimes when I stop the pocast, Siri will then work, but then I cannot get back to the podcast from the headset. I have to pull out the phone and restart the podcast playing from within the Podcast app.