iBooks files and ePub files work very differently: which you should use in iBooks Author really depends on what type of book you're making, which devices you're targeting, and what features you'd like.
iBooks files are primarily targeted at iPads, but can also be read on the Mac and iPhones. They're the ones listed as "Portrait" or "Landscape" in iBooks Author. iBooks format is more for things like coffee table books, image heavy books, or textbooks: books designed to have lots of elements like photos, videos, interactive elements, etc. that work almost like a webpage vs. just being an electronic version of a paper book like ePub. The reader won't be allowed to resize the text in iBooks files as it would throw the careful layout of all the pages and their photos etc. completely out of whack. Honestly, I think font resizing could work okay with iBooks files if done carefully and I wouldn't mind seeing it as an option in the future, but I see why Apple limited it. So choose your font and font size carefully with iBooks: ideally optimizing for a regular sized iPad, but keep in mind an iPad Mini will show it slightly smaller and an 12.9" iPad Pro will show it slightly larger. Mac users can just resize the window to make text bigger, and iPhone users can zoom the page (reading these types of books isn't ideal on an iPhone but hey at least it's offered).
ePub is much simpler and meant more for a simple book like a novel that's just words. For these the reader can choose their font size as there is theoretically no layout to be messing up with a purely text book. ePub is useful beyond Apple devices for things like Kindles.
In theory you *could* write a novel as an iBooks file, and you *could* make a coffee table photography book as an ePub file, but neither is ideal. If your book is just a novel or has only very simple image elements (like a single photo at the start of a chapter), I'd go for the simplicity of ePub. If you want to make something more complicated than that, I'd start with an iBooks file but keep in mind you won't be able to use that file outside of Apple Devices: getting it onto a Kindle will require you to "remake" the book in some other software, and also to dumb it down a bit since iBooks has many features ePub does not.
