How large is your Photos Library? Upgrading a very large Photos Library may take many hours without any visible progress, if the library is large. In that case let it run.
Is your external drive compatible with macOS 11 Big Sur? A Photos Library must not be on a volume, that has been used for Time Machine backups or is accessed over the local network. If the external drive has not been prepared as described here (Move your Photos library to save space on your Mac - Apple Support) move your Phots Library to a different external drive, that is locally mounted and correctly formatted, then try again.