Photos (Catalina 10.15.7)

If you change the first box in conditions to Album, change the second to "is not" and the third to "Any" then the smart album should only contain photos that are not in any album. That works for me to do what you are asking.

You sure about that?

Create a smart album -- Photos File menu > New Smart Album and set the condition to Album - is not - Any.

Your options change when you change the first term: so those are the Photo options, If you select 'Album on the left you'll get different option on the right...