It should update the existing book if you are working from the same file and not resetting the internal GUID of the book.
Check this article link below to understand how books have a unique ID.
https://support.apple.com/en-us/HT202692
Create a new book in iBooks Author with the same look and feel of one you created previously - Apple Support
Also, you'll notice the book version behave differently depending on if you:
1) Proof from computer doing preview - Shows Proof banner. Books "previewed" from iBooks Author should always show the Proof banner, even after you have opened and viewed the book.
2) Adding by double clicking the .ibooks file - Shows New banner
- If you double click a new version of the book it will not update
- If you DRAG a new version into the iBooks app it should, and should show the following popup. The New banner disappears after you have viewed the book.
If you select "Replace" it should update the book to the latest version.
If the book were on the iBooks Store, the iBooks app would indicate a new version is ready to download, as shown below.
Adding version information or date on the cover or inside the book may help to easily know which version you are on.
I hope that helps some.
