I think the answer has already been stated (at least for most users) it's usually just a small hair or something similar.
I've been using iMacs for over 8 years now and have had a Magic Mouse do this off and on for both my original machine, as well as my new one I'm presently using. I also have 3 large and very hairy pups. German Shepard/Siberian Husky mix, and because of them, I have excessive hair, dust etc. that we try to keep to a minimum, but it's unavoidable and impossible to completely control. ... So when the mouse gets jumpy, I can usually just blow a little air in the hole where the sensor is and it's fine. (Because it removes some sort of debris) But sometimes it takes more then just puffs of air and I've had to remove the cover (as others have stated) to get something that wouldn't remove with just air. (Unfortunately, there isn't a cover on the newer Magic Mouse II, which is simply plugged in via a lightening port, since it has re-chargeable batteries. Though the port's location could have been better thought out. IMO) If you ever happen to get something down in there where the puffs of air won't remove by itself, you might need a bright light and magnifying glass to check it thoroughly to see what you missed, which is usually a small hair or similar. - (The Magic Mouse II can be taken apart as with anything, but it's not designed to be and a simply look on-line to see how it's done will probably keep most from attempting it)