Somehow a third party tried to remotely take control of my mouse!

I have an iMac (10.6.8). Today someone tried to take control of my mouse to open Finder, a youtube downloader program, etc. I caught this happening both times and fought for control of the mouse until I ultimately had to put the iMac to sleep to prevent them from getting into anything personal, etc.

Following the first round, I changed my password to my imac and macbook, made sure the firewall was set up, disabled making either discoverable via bluetooth, and somehow it happened AGAIN. For a final test, to see if this happens a third time, I disabled the remote control infrared receiver.

Can someone tell me what the BLEEP is going on (outside of the fact that someone has managed to gain remotely access to my computer) and what I can do to stop it. I am in teh process of compressing my files and moving them off the imac and doing a full system reinstall to set it back to factory settings when i first got it and see if that does the trick. I am very careful about what I download so I dont think it was anything like that (mainly because I use my iphone for email)

Ideas? Thoughts? Suggestions?