My Mouse is Going Crazy!

I wish I knew what the heck is going on with this "magic mouse" I've got. I have no extra mouse gestures selected, no shifting tabs, none of the other ones, only the scrolling ability. Yet suddenly the page I'm on has turned into a different one! And when I try to get back, by clicking the back button, it just takes me to the Start Page! What is going on? I click somewhere, and the page jumps halfway up the window, sometimes trying to click an app in the dock requires me to chase the icon as they keep moving around. Or I click on one line or word, but instead another one will respond.

And speaking of responding, sometimes NOTHING will happen, clicking does nothing, sometimes I don't even know where the pointer is, no matter what I do, shaking it, clicking it, no, nothing. This will sometimes last a few minutes, other times, 20 seconds. Oh, and then sometimes, the pointer literally just floats around the window, all on its own! Is this a bit outside of typical behavior? If it were only this mouse, I'd just go get a replacement, but with a generic mouse, some of the same things happen, just not any of the gesture movements, but it will once in a while just stop responding, or not be very accurate when clicking... so it may be a deeper issue, or it could be primarily the magic mouse, as a lot more weird stuff happens when I use it.