Sorry, the Photos app on an iPhone does not work that way.
You can add photos to albums till your heart's content, but you cannot remove them from the main "Recents" album
"Recents" contains all photos taken and added to the Photos app always.
To avoid potential unnecessary duplication of photos and usage of storage, Apple only adds a pointer to the photo in the albums you add them too. This allows you to have the same photo in multiple albums without multiplying the storage used by them.
You don't need to scroll through thousands of photos. Apple's photos search is power full enough to find things and text within the photo easily.
There are also options to see the photos by date or location if required to make it easier to find things in the "Recents" album
Once added to an Album, you can find your photos there and search within the album too.
click here ➜ Organize and find your photos on your iPhone, iPad, or iPod touch - Apple Support