You will have to keep the external drive connected to your Mac at all times, even if you are not using Photos, if the large library is your System Photos Library. Photos is expecting your System Photos Library to be always available, as long as you are logged into your user account, the connection to the Photos Library might get lost. This might be inconvenient, if you are moving the MacBook Pro around a lot. Or ist the Mac continually resting on a desk and you never have to carry it around?
As far as I can tell, the library size matters particularly when upgrading or updating the system, because Photos will have to scan the library again. when some algorithm has been improved by the update. The library size matters, when we need to repair the library, also for the "curating of best photos" background processes. These are processes, that will settle down, once they have finished the scanning, but will initially make your Mac slow.
One item will always be slow for a large library. You may want to keep the number of smart albums small, as they are depending on all photos in the library and need updating, whenever a photo is edited, tagged, removed or added.