Photos has intentionally been designed this way.
There are two flavours of albums in Photos - the albums we create ourselves ans the albums created by photos for us.
- The albums in the Library part of the list of albums are populated automatically by Photos for us, based on all photos in the library. Each of them is showing us a selection of photos according to rules, specific for this album and has a fixed sort order. the only way to change these albums is todelete the photos from the library, and that will remove the photos from all albums. Use the Library as the foundations to build your own albums on top of them.
- We can define folders and albums and sort the items in tour own albums freely. we can remove photos from our own albums without deleting them from the library.
You may want to try this work-around: To achieve what you want to do, create your own album "My Recents" and add all recent photos to this album. Then use "my Recents" as the starting point to distribute your photos and remove the photos from my Recents when you are done.
Or use the Favourite Heart ♡ to tag the photos you have added to an album, so you can see at a glance, if a photo is already in an album. This will work, if you only want to add Favorites to albums, as I am doing - I want to keep the albums small and manageable, particularly on an iPhone with the small display.
Previously, it has been easy to remove photos from the Library albums like Recents, without deleting them outright. We could simply hide photos we already had added to an album. They would remain visible in the albums, but be hidden from the library. But many users complained and wanted to use the "Hide" button to keep confidential photos secrete. Finally Apple changed the way hiding is working. We can now keep confidential photos secret by hiding, but have lost the ability to remove photos temporarily from the Library section, but keep them in the albums.