How can you determine all photos not in a Smart Album?

You cannot.

If you could define a smart album asking for all photos not in a smart album, it would be a logical contradiction, because the photos would be in a smart album, and then should not be in this album.

Photos does not support this.

You can only open all your smart albums, one by one, and add the photos from these albums to a standard album "XXX". Then create a smart album with the constraint "Album is not xxx".