If you're using iCloud Photos, then the best bet is to make a new Photos Library, set it as the System Library, and let it load the pictures from iCloud-- then you know the Library is clean.
Is your Library being restored to your User's Pictures folder?
If you're not using iCloud Photos: is the Time Machine backup also from Tahoe? If not, then Photos will have to spend some time upgrading the Library to Tahoe.
If the Library is already Tahoe, you may need to try these things:
- Use another library if you have one, or make a small test Library with a few pictures in it and see if the same thing happens. You can create a new Library by closing Photos and then option-clicking on the Photos icon.
- Rebuild your Photos Library--close Photos and re-launch by option-command-clicking the app icon. This can even take a couple of tries. It's good to have a backup. If you use iCloud Photos, it’s possible that this will cause a re-sync that lasts long enough to make you nervous.
- Make a new user and see if the same problem recurs with it. This is a bit harder, because a new user can't access a different user's stuff. You would need to temporarily move your Library out of your own user's Pictuers folder up to the general "Users" folder.
The idea is to find out if the problem is with the Library, with your account, or with caches and login items, and stuff like that. One of these steps may fix the problem, but their failure to work also helps figure out what's happening.
Let us know what happens…