Hi rcsmit,
It's been a while since I finished the game, but I still have it. The copy I have is for both Windows and Mac, noted on the side of the box at the bottom. Does yours say the same?
An .exe file is a Windows executable (application), and so will do you no good without a Windows emulator such as Virtual PC.
The box should contain an oversized CD jewel box that holds four CDs. When you insert disk one, you should be presented with this list of files:
Desktop Folder (folder)
Data (folder)
Myst3 Installer
QuickTime Install Cache
QuickTime Installer
Readme.txt
The file "Myst3 Installer" should have a VISE icon. Use that to install Myst 3 on your Mac. The initial installation only works with OS 9.1 (Classic). In order to use it in OS X, you need to download two files from
here. Which is a link within one of the links ali brown gave you. On the page I linked to here, you'll see two files under the heading "File Attachments". They are:
M3OSX1.sit (A shell which allows Myst 3 to run as an OS X program. Install this first after the initial installation of the game)
m3patch122
usmac.sit (Updates Myst 3 to the last patches released by Ubi Soft. Install this last)
You should be ready to go then.