The usual workflow is, to select all images in album first, and only then to create the book using "File > New Book".
This way you will see all images in the filmstrip below the book in split view and drag from the filmstrip to the book pages.
If you want to add more images from the album to the book, select them while browsing your album and drag them to the book icon in the library inspector panel.
But if you really want, you can have the book and the album open in a browser window at the same time, see Working with Two or More Browsers Open
- Select the book in the Library Inspector panel and switch to split view.
- Then ⌥-click your album.
Both your book and your album will be visible in the film strip.
Two browser windows open in the filmstrip (from the user manual).
I learned this trick from Kirby Krieger - thanks, Kiby!