This sounds like it may be one of the gestures on the Magic Mouse, "Swipe between full-screen apps".
Place two fingers on the surface of the mouse and move them from left to right. This will move you to the dashboard desktop on the left. Sounds like you're moving part way there accidently. Lifting you fingers will let it slide back.
If you'd like to disable this gesture then go to System Preferences > Mouse > More gestures tab > uncheck Swipe between full-screen apps.