So you are using iMovie just to transcode them. Apparently .mod files are actually mpeg and in some cases can be made 'acceptable simply by changing the extension to .mpg though this does not always work.
Have a look at the contents of your iMovie 10 library (right click > show package contents). In each event folder there will be an 'Original Media' folder. Do these files still have .mod extensions? If not you could try importing them into iPhoto/Photos, avoiding the need for exporting them.
Otherwise you need to find a way of converting them into a format that can be imported directly into iPhoto/Photos. If you google this you will find a load of paid third party apps that claim to be able to do this but you may be able to use free apps such as Mpeg Streamclip.
Maybe someone else more familiar with old movie formats can chime in here with details.
Geoff.