It is bug in Catalina: You have to remove any photo from an album, before you delete the album.Otherwise the photos will always be counted as being in an album, see:
Problems with the Smart Album Rule: Album is not Any in Photos 5.0 - Apple Community
You could try to use an AppleScript, to find the albums a photo is in, but Apple script is more complicated on Catalina when scripting albums than before, and very slow. I cannot early recommend to use this script, unless your library is very small or you are very patient: How to find out the albums a photo is in: Jacques Rioux's Script, Catalina Version for Photos 5 - Apple Community
I am mostly using smart albums based on keywords. Instead of adding a photo to an album I tag it with the album name as a keyword and the photos will appear automatically in a smart album with the rule "keyword is ... ". This way I can see in the Info for a photo all album names as the keyword.