There is nothing special about 3.7 GB of memory versus 4 GB in Mountain Lion. The OS is fully 64 bit and doesn't care one bit; it's just that the 32 bit application will see a 4 GB address space.
It could be that there are other issues besides just a memory leak. Perhaps the application has something that is blocking the main thread. That is the main reason for the beachball: The main thread is blocked for some reason. If you switch to another application does the beachball continue?