With only 4GB of RAM, how much do you have devoted to the virtual machine running under Parallels? Even if it's a small amount, it's going to make page outs and swaps used very high. From my perspective, 8GB of RAM is the 'sweet spot' for running Mountain Lion.
Other than upgrading your RAM, the only thing that you can do to speed up your machine is to install a SSD. You can also check the SSDs at Crucial and OWC and decide if one is right for you.