Can I retrieve photos that were not backed up after a factory reset?

The backup is not the best way to deal with photos anyway. Photos should be imported to the computer the same way you would import photos from a digital camera. You should always do that before doing any software work on the device, since you could always have a failure with an update. The iPhone does not have a recycle bin.

Have you sign into icloud on your phone and see if there is any backups? Icloud keeps the last 3 backups.

Have you tried restoring from your icloud backup iOS: Back up and restore your iOS device with iCloud or iTunes

No no and no, since you didn't back them up to iTunes or icloud.

The full message you say probably stated, that you had to much data to use the free 5GB to backup your phone.

You would need to delete some stuff off your phone in order to backup or buy more backup GB.

Also you should be importing your pics to your computer as you would a digital camera.