Photos in external drive son't sync appropriately

MvanTTheThird wrote:

...

it is almost as if the external drive must always be connected in order to sync correctly. Once it is disconnected en reconnected the downloads from iCloud that were synced to iCloud earlier don’t sync to the external drive.

...

Afaik, your system photos library needs to be always available while you are signed into your user account. The System Photos Library is special. Make sure the external drive is connected and mounted before your sign into your user account. Otherwise the Photos and iCloud background processes may lose the connection to your System Photos Library.

Is the external drive correctly formatted and prepared as described here? Move your Photos library to save space on your Mac - Apple Support

Anyway, albums are taking a long time to sync from iCloud to the Mac. I noticed on my Macs that the albums have only been updated once all photos have been synced after the upgrade to macOS 11. which system version is installed on your Mac? Are you asking about standard albums in Photos or Shared Albums in iCloud?Share photo and video albums with iCloud – Apple Support (UK)

Shared albums are shown in your System Photos Library, but are not a part of the System Photos library. Whenever Photos is losing the connection to your System Photos Library it has to download the shared albums again from iCloud.

If they share properly when located on the primary disk then there's the issue. It could be that some energy saving or sleep feature is affecting the drive being detected. I also get the impression that keeping media libraries on external drives has for a long time sometimes had issues. It wouldn't surprise me if the Apple solution would be for you to subscribe to more iCloud storage and keep everything there.