It takes a very long time for the modification date and the size of the Photos Library to update, even with the library in the Pictures folder in your home folder. I have seen the same date and size there for months, in spite of working with my photos every day., and my library is on an APFS volume in my Pictures folder The internal folders inside the library package are showing the correct modification date, while the library is now showing a modification date for August 31. I am ignoring these date and the size since the Catalina upgrade.
But the long time to open the library seems to be a serous problem. How is the external volume connected to your Mac? Is it directly plunged in or accessed remotely? You need a locally mounted volume.
Has the external volume been used by Time Machine? Then you cannot use it for a Photos Library. As a test, try to set the "Ignore Ownership on this volume " flag for the external volume as described here: (Move your Photos library to save space on your Mac - Apple Support). If you cannot set this flag, the volume cannot be used.
The external volume should always remain connected to the Mac, as long as you are signed into your user account, even if you are not working with Photos. The background processes will try to access the library while you are not using Photos, and if you disconnect the volume and plug it only in, when you want to work with the library. it will slow Photos down.