I can't go back and just delete those few photos from the original album without also deleting them from the album I just moved them to
That's not true. If you select a photo in an album and use only the Delete key it will be removed from that album only.
However, lets make it very clear that when you say "original album" you do mean an album and not the Moment that the photo gets added to. If you delete from an Moment then it will be deleted from everywhere in the library.
See the Note at the end of this reply.
Deleting Photos from a Photos Library
1 - regardless of where you are in the library, i.e. Moments, Collections, albums, smart albums or projects, select the photo(s) you want to delete and use the key combination of Command+Delete to move the photos to the Recently Deleted smart album.
2 - use the File ➙ Show Recently Deleted menu option to go to the Recently Deleted smart album.

3 - in the Recently Deleted album you can opt to let the photo automatically delete after the 30 day waiting period is up, delete them immediately or restore them to the library.

NOTE: deleting a photo from an album, slideshow, book, etc., with only the Delete key only deletes that photo from that item. Deleting a photo from a Moment or Collection deletes ALL occurences of that photo in the library.
