Magic keyboard is not discoverable and only connects when plugged in to iMac

My magic keyboard suddenly stopped working the other day. I thought it was because of the battery. Charged it (through power plug) for a bit and still not working. However, after I plugged it in directly to my iMac it worked and checked the battery level, there was none. The battery level didn't show up at all on the keyboard settings. Not even 0%. It was just gone. The keyboard still showed up under the bluetooth setting.

Tried to turn it off and on for several times and nothing is happening for the bluetooth connection. I deleted it from the bluetooth setting and because it wasn't discoverable, now it isn't listed on the bluetooth setting anymore. However, once I plugged it in again and turned it off, the keyboard magically still working. So, the only way to use the magic keyboard is by plugging it in directly to the iMac, which is fine by me as long as it's working, but still bugging me. Why the bluetooth suddenly stopped working one day? Couldn't find any way to fix it. Have tried the steps from this thread Magic keyboard only connects when plugged… - Apple Community and still not working. Mainly because I couldn't even turn my bluetooth off. It refused to turn off no matter how I kept clicking yes to turn the bluetooth off.

Anyone have the same problem? Just to highlight the problem, the keyboard is working perfectly, regardless the switch was on/off, as long as it plugged directly to my iMac. So, I'm pretty sure the bluetooth connection is the one that needed fix.