Are your Library and your Masters on the same drive?
Have you stopped using iPhoto (stopped, as in "never use it")?
An easy thing to do is to consolidate the Masters for all your Images into your Aperture Library. Go to Photos view, select all Images, and run "File➞Consolidate Masters". Your Library then contains all your Images and all their Masters, so you can delete any image files not in your Library without worrying about deleting Masters (NB: but make sure there are no image files outside your Aperture Library that you want to keep).
After you've cleaned house, you can relocate your Masters wherever you would like them. Aperture makes it easy to move them: "File➞Relocate Masters".
Many users want a logical Finder folder structure for their Masters. This is also easily done, but one is somewhat limited in one's choices. Keep in mind that a logical Finder folder structure, while comforting, is of no use within Aperture, and does not make Aperture perform any faster.
When you use "Delete Version" on the only Version in your Library based on the Master that the Version you are deleting is based on, Aperture deletes the Master as well (or asks you, in the case of Referenced Masters, when you empty the Aperture Trash).
This is spelled out, I hope with some clarity, in this recent post.