I have an iPhone 6S, with OSX 13.3.1. I have had the same problem where, Photos eats up 50% and more of battery usage, even when I am not using the photos app during the time period I'm seeing reports for.
I noticed this started soon after I started using the "Albums" ... the Shared albums feature. The Photos app really became started performing poorly, it would crash when trying to work with the albums, whereas, it never crashed before, the phone started getting hot, the battery dropping like mad, etc.
Recently I got really fed up with it, anyways I downloaded everything I could from the shared albums, and disabled the shared albums. After 3 days it seems to have solved the problem. To disable:
Settings > Click on your face, the top Apple ID menu > iCloud > Photos > Turn everything off, especially "Shared Albums."
The photos themselves are backed up I believe as part of the backup of the phone. Periodically I move all the photos off the phone, and I have my own methods for simply backing things up using multiple external hard drives. I believe this is the best, most flexible, cheapest, safest way to do backups.
Thanks! Good luck,
Kevin