My 27" iMac running Catalina with an external monitor always comes out of sleep a couple of clicks brighter. How to fix?

My year-old 27" iMac running Catalina with an external monitor always comes out of sleep a couple of clicks brighter. This is really frustrating because I calibrate my monitor for photo work so it's very time-consuming. My external monitor stays calibrated but the 5k iMac screen is always brighter after sleep. Any ideas on how to fix it? Thanks!