You need to remove Safari 4 before you can install a previous version, but you can't just delete the application and expect to be able to install something else. You have to do this:
Download
Pacifist.
Go to /Applications and delete Safari.
Go to /Library/Receipts and delete the Safari403Upd.pkg file.
Open the Safari installer you previously downloaded (3.1.2, or 3.2.3) with Pacifist.
Select the top-level package icon in the Pacifist window for that installer and click Install in Pacifist.
When asked by Pacifist, choose to overwrite or replace all files that already exist.
Quit Pacifist when done, and restart your Mac.