While this support article (http://support.apple.com/kb/HT5068) discusses media in the context of an HTML widget, you could try converting the video with QuickTime Player, Final Cut Pro, Compressor, etc., to the same format, i.e.:
- H.264 video in an MPEG-4 container with an ".mp4" file extension.
This way, iBooks Author may not perform an automatic conversion, and you can control exactly how the video is converted.
Hope that helps!