Is it possible to set the font face when you insert an equation into iBook Author using MathType

I don't believe so, no. If it is possible to style the equations, you can do it with CSS. Even though you may insert equations with MathType, once they're in your iBook, they'll be represented with MathML, not with an image like you may be used to seeing in Word or Pages.

iBooks Author's CSS is sometimes nonstandard, but try this for the equations

math {font-family: Arial, "Lucida Sans Unicode", Verdana, sans-serif;}

substituting your font(s) of choice. Let us know if it works; I'd like to know but can't try it out at the moment.

Ok, I had a chance to test this, and was able to verify the above style does allow you to change the math font. BEWARE: Not all fonts have all the necessary glyphs to properly display math notation. Try several fonts and font families and decide which ones work best. Whatever font(s) you choose, it makes sense to embed them into your iBook (see http://www.macworld.com/article/2013246/hands-on-with-ibooks-author-2-0.html).

So where does the math CSS go?

Within an iBook, each section has its own CSS. Export the book to a location on your computer. UnZIP it by changing the extension from .ibooks to .zip, then unzip it to your computer. Within the book, in the OPS folder, each section will have a different file named content1.xhtml, content2.xhtml, etc. There will be a corresponding CSS file in the assets/css subfolder. Open as many of these (content1.css, content2.css, etc. -- you don't need ones like content2-flow.css) as you need, and add the math style to the file as a new line. After you save the CSS file(s), change the extension of your book from .zip back to .ibooks.

There are no doubt others who monitor this forum who know more about this than I do. I'd be interested in hearing from others, because I'd like to know more about it myself.

Hope that's of some help.