the photos app's display problem of macOS Big Sur

Create a new user account and see if the issue also occurs in that new account. Let us know. This is important to discern if it is a system wide problem or specific to your account.

Also Please run Etrecheck from your normal account and post its full report here.

Use the “additional text” button and paste the report into the text box.

This will give us information which may help figure out the cause of the problem.

FWIW this is the first time I have seen this which leads me to believe that it is something in your mac rather than a bug that would affect a lot of people. Otherwise we would surely have heard of it by now.

You might want to try AppleCare. That may be a better way to highlight the problem to the developer's.

I can't replicate what you are seeing, so the developers may have no idea this is a problem, or it could be isolated to something you have installed/your particular setup. I can only find one other report of the similar problem from when Big Sur was released.

Are you using multiple monitors? Are they set up to have separate spaces or extend into a single display?

I have two monitors connected to my Mini running Catalina and I see a similar think happen when I enter Time Machine. However, both displays are supposed to go black in Time Machine. My monitors are different shapes, and the desktop on the non-Time Machine has the cut off appearance you are showing, but no graphics tearing at the edges.

May want to send some Feedback to Apple

Did you try in a new user account?

That Etrecheck report may really help. I am quite confident that there is some conflict with third party software behind the problem.