External SSD Library Photo don't always work

How is the external SSD connected to your Mac? It needs a wired connection, not a network access.

And even with a wired connection, the external drive must not get ejected, while you are logged into your user account. Are you occasionally getting warnings about the external drive being ejected, when your Mac is awaking from being asleep? If that happens, Photos will lose the connection to your System Photos Library, because the System Photos Library needs to be available at all times, while you are signed into your user account.

So log off, before you eject the external volume.

To avoid this problem, I am keeping my System Photos Library on the internal drive and are keeping it small enough, so it will fit, and the larger libraries on the external are fine, as they are not the System Photos Library, and I am switching back to the library on the internal drive, when I have stopped working with them.

Have you tried repairing the library by launching it with the Option+Command keys held down?

Also have you installed and run any "cleaning", "optimizing", "speed-up", anti-virus or VPN apps on your Mac?

Create and log into a new, basic, admin user account, Set up users, guests, and groups on Mac,  and see if the problem persists. This tells us if the problem is limited to your user account or is system wide.

Note:  do not launch any 3rd party apps in the account so as to not install any of the supporting files that most create.

Do you have "Ignore ownership" enabled on the external SSD?

Well, first let's not go again the old adage; Don't 🔭 a 🎁 🐴 in the 😮.

Some times when a user is having a problem with a library if they create a new library import a photo or to into it and then go back to their problem library it works. It may have to do with cache files. Doing it from another account may be similar. Glad you got it worked out.