Usually (always?) macOS & Photos upgrades updates also the Photos library (at least if the user opens it).
You could restore the old Photos library from a backup. You do have a backup?
Or install Monterey on another internal or external volume, export images and movies out and import to a Big Sur Photos library. Yes, that is a major hassle.
Or bite the bullet and upgrade back to Monterey. Last autumn I reverted to Big Sur from Monterey (despite carefully pre-testing all my mission critical apps on an external Monterey volume. But I digress...) but I am very soon going back to Monterey. Thanks to Carbon Copy Cloner backup(s), this was quite easy and fast.