How do I stop the proactive assistant from showing the music controls on the lock screen?

Since updating to iOS 9 on my iPhone 4S, it has been trying to predict when I listen to music and has been putting the music controls in place of the large clock on my lock screen. The problem is that I don't listen to music on my phone very often, and these controls are making it difficult to tell the time at a glance. I would like to stop the controls from showing unless I'm actually playing music, if possible.

Does anyone know if there is a way to turn off the proactive assistant music controls?