If you delete a photo from an album, it will be deleted everywhere. If you remove a photo from an album, it will only be removed from the album. The "delete" key ⌫ is sometimes bound to "Remove from album" and sometimes to "Delete Photo". It depends on where you are viewing the photos.
Which album are you viewing, when you are trying to remove a photo? Some albums do not allow to remove photos - for example the smart albums or the albums automatically created by Photos. here we can only delete photos from the library but not remove them from the album. These albums are populated automatically based on some rules, and to remove the photo from the albums we would need to change the rules, which is not possible for the automatically created albums.