Until Apple offers a solution, here's what I do instead of restarting. When you notice Enhanced Dictation has stopped listening, paste the following command into the Terminal application (found in /Applications/Utilities/) and press return:
killall -9 DictationIM com.apple.SpeechRecognitionCore.speechrecognitiond com.apple.SpeechRecognitionCore.brokerd
This will force quit the Dictation UI and the low-level speech recognition processes. You should see the microphone menu bar icon disappear and then re-appear (if you're using keyword mode) and after a few seconds be ready to listen once again.