Unwanted music controls on startup screen

Recently sometimes (but not always) when I turn on my iPod touch, below the time and date on the startup screen are audio controls (Play and volume). I realize that it's trying to be helpful by allowing me to quickly resume the music I was last playing, but I rarely play music on my iPod, so all it's actually accomplishing is to partly cover my lovely photo of Bruce Wayne. 😉

I went to the Music app but couldn't find any option to tell the iPod, "forget about playing any music." It says that I have a paused "Now playing" track, and all it will let me do is choose another track, not turn off the music completely.

I've only had this iPod for a few months. It seems unlikely (but I guess it's possible) that I never got this stuff on my startup screen previously because I had never played any music on it. Is there any way to make it go away?