How big can a photos library be?

If you just merged many photos into your library, Photos may still be busy with the background processes to scan the photos for faces and categories. Keep the Mac powered and running over night, so Photos can finish the background processing. Keep the drive with the Photos Library plugged in, even if you ar enot using Photos, until all photos have been scanned.

Is your external drive compatible with Photos? The file system format needs to be MacOS Extended (Journaled), not case sensitive, and the drive needs to be connected, whenever you are logged in, so the background processes can run. https://support.apple.com/en-gb/HT201517

And it must not be a NAS.

I have not found any indication, that a smaller library will improve the processing, as long as there is no storage problem. The only problem I noticed wit a large library are smart albums. If you are using too many smart albums that need to scan the complete library, Photos may be slower. I am trying to keep the number of smart albums below 100.

Force quitting can damage the library. In that case I would make a backup of the library and repair it, if you are now experiencing issues with the library.

https://support.apple.com/en-gb/guide/photos/repair-the-library-pht6be18f93/mac

Apple did not publish a size limit, but since Photos is supposed to be able to open Aperture Libraries and iPhoto Libraries, it should be able to handle libraries with 1000000 photos, if the drive with the library is large enough and you have enough free working storage on your system drive.

VERY slow and unstable.

Which system is running on your Mac? Photos is running very well for me with a library with rough 50000 photos on a MacBook Pro (Retina, 15-inch, Mid 2015) and macOS 10.13.4.

Some users reported that Photos is slow or crashing when using the retouch tool. Is that when you are seeing the lagging and crashes?

Can you tell us more about your Photos Library? Is it on your system drive or on an external drive? Are you using iCloud Photo Library or optimize storage? When did the problems start? After you upgraded software? After you imported many new photos or videos?