low/high modes alters results from Pytorch based ML model on M2 Max Macbook Pro

So i seem to have a situation where the actual calculations involved in a machine learning algorithm depends on whether or not i am in low/high mode.

I started a machine learning training run before going to bed. In doing so I switched the machien to low power mode and left it plugged in to ac power. In the morning i then unplugged it and since it wasn't set to use low power mode on battery i think it switched to high power mode.

I have two plots, one shows how long each training iteration took over the 60 or so training iterations it did over night. You can see that the time it takes to do an iteration starts low (because it was initially in high power mode) then rises (because almost immediately after starting the run i switched the machine to use low power mode). Then at 9.03 am when i unplug the machine the time it takes to perform a training iteration falls again (because it's now in high power mode again).

So far so good but what I am very confused by is this second plot of the error associated with each training step:

This would seem to show that when the machine switched to high power mode at 9.03 am actual model behaviour changes.

Is this a bug? Expected behaviour? Indication of a hardware issue?

Note this is using, miniconda with pytorch 2.0.0 cpu_py310h32bc11d_0 and transformers 4.27.4 pyhd8ed1ab_0. With a 64 Gb Macbook Pro with M2 Max on OS X Sonoma 14.0 (23A344)