If you know or suspect that a hostile intruder has either had physical access to your computer, or has been able to log in remotely, then there are some steps you should take to make sure that the computer is safe to use.
First, if there's any chance that the incident will be the subject of legal action, then you should do nothing at all without consulting a lawyer or the police. Your computer would be the principal evidence in such a case, and you don't want to tamper with it.
Running any kind of software to scan for "viruses" or "rootkits" is worse than useless. If I broke into your system and wanted to leave a back door, I could do it in a way that would be undetectable by those means—and I don't pretend to any special skill as a hacker. You have to assume that any intruder can do the same. The "anti-virus" software itself will slow down and destabilize the computer with no offsetting benefit.
The only way you can be sure that the computer is not compromised is to erase at least the startup volume and restore it to something like the status quo ante. The easiest approach is to recover your entire system from a backup that predates the attack. Obviously, that's only practical if you know when the attack took place, and it was recent, and you have such a backup. You will lose all changes to your data, such as email, that were made after the time of the snapshot. Some of those changes can be restored from a later backup.
If you don't know when the attack happened, or if it was too long ago for a complete rollback to be feasible, then you should erase and install OS X. If you don't already have at least two complete, independent backups of your data, then you must make them first. One backup is not enough to be safe.
When you reboot after the installation, you'll be prompted to go through the initial setup process for a new computer. That’s when you transfer the data from one of your backups in Setup Assistant.
Select only users in the Setup Assistant dialog—not Applications, Other files and folders, or Computer & Network Settings. Don't transfer the Guest account, if it was enabled.
Reinstall your third-party software from original media or fresh downloads—not from a backup, which may be contaminated.
Unless you were the target of an improbably sophisticated attack, this procedure will leave you with a clean system. If you have reason to think that you
were the target of a sophisticated attack, then you need expert help.
That being done, change all Internet passwords and check all financial accounts for unauthorized transactions. Do this
after your system has been secured, not before.