Auto brightness and screen dimming going crazy

I've got a M1 Pro MacBook Pro 16" running Monterey. I like to leave auto brightness on and it seemed to work just fine until I decided to turn on the "screen dimming when on battery power" option. I quickly realized I wanted to disable just that, so I did, but now the screen keeps dimming and "undimming" each time I plug it or unplug it and the automatic brightness doesn't seem to work anymore. The screen just gets stuck in whatever level of brightness I leave it in, it goes up if I put up some heavy light over it but then it goes back to the last place I left it, instead of moving according to ambience light.

Example: I tested it manually leaving the brightness barely legible in a rather low light room, then put the iPhone flashlight on it. It moves almost all the way up, but when I turn the flashlight off it goes back to barely legible. Before this it used to "measure" the light in the room and just dial it in automatically, but apparently It's not doing it anymore and I have to manually move it every time the ambience light changes. It's pretty annoying.

BTW the system is updated and I tried restarting the machine but nothing changed...