Is there a way to only backup newly modified items? Is something like this only done in time machine?
I would use cloning software to update the copy of the library, for example Carbon Copy Cloner. Cloning software can make incremental backups. And include the library into the regular Time Machine backup. One backup is not enough.
Photos should be able to cope with a large library. But the basic maintenance will take a long time - updating the library, copying it for backup, repairing it. And do not allow the volume with the library to get too full. a large library will need a lot of free working storage for updating and repairing.
The user interface of Photos has not been designed for a large library. You may get impatient, because some views cannot be sorted to make it easier to search for items. For example, we cannot sort the People album alphabetically, or group related people into albums. The list of keywords cannot be structured. The list of albums in the "Add to album" dialog cannot be sorted. There are not many tools for batch processing. We cannot stack photos or compare photos side-by-side.
It is possible to use photos, even with a large library, but we have to invest some work into structuring the library with folders, albums, keywords, and create or own interface for accessing our photos and videos. Here are some ideas, how to design our own access structure, to make working with Photos easier:
New to Photos? Some Considerations on How to Design your First Photos Library in Photos 5.0
Which system version do you have installed? Your older videos may not be corrupted, just incompatible with the new system version. The upgrade to Catalina made some older videos and images files incompatible, and the background Photos may be wasting a lot of time when trying to process the incompatible media over and over again. For example, Photos 5 on Catalina can no longer process PDF files.
before upgrading to Catalina the older items should be converted to a compatible format.
I have been using this list for iMovie as a guide line what to look for in Photos before the migration to Catalina: About incompatible media in iMovie for macOS - Apple Support