Unable to remove/re-add audiobooks on iPhone 13 in iTunes

I am unable to remove audiobooks from my device in itunes. I can delete audiobooks directly on my device, but they still show in iTunes on my device - even if they have been deleted from the main iTunes library. Once a book has been deleted from my device, I can’t then re-add it. The only solution I’ve seen is to reset factory settings - which seems absurd. I have an iPhone 13 and fully up to date OS.