How can you delete a book?

If you've downloaded into the iBooks app, and not already done so, then you should be able to delete it from the bookshelf - can't remember how it was done on iOS 5.0.1, but is there an Edit or Select button at the top of the bookshelf ?

To hide it from the cloud (you can't permanently delete any purchase) so that it doesn't show for redownloading : Hide and unhide purchases in iTunes or iBooks on your Mac or PC - Apple Support.

We are, but your tagline in your first post says that you are using iOS 5.0.1

If your phone is on iOS 9.1 then you can delete downloaded books from the app via the Select button at the top right of the bookshelf.

To hide them from the cloud for redownloading the link in my above reply still applies