This is the manual for the two-battery variant of the board:
https://manuals.info.apple.com/MANUALS/1000/MA1208/en_US/Apple_Wireless_Keyboard_2009_UG.pdf
There were three variants of that keyboard model. One used three batteries and the others take two. Fortunately the board has a nearly-hidden icon that shows how many batteries and their orientation
Flip the board upside down and look at the tubular portion:
The icon indicates this example takes two batteries.
The difference between the two-battery and three- battery variants has caused a lot of people to think they had a battery stuck but had a two-battery model.
If shining a light in the battery compartment clearly shows another battery and not the normal metal contact, you have to get just a little physical. Normally holding the board with the battery compartment pointed down, then light tapping the keyboard on a "slightly hard" surface will free a stuck battery. I tap them on whatever magazine is close to to my workstation.
If that does not free the stuck battery, I bump the sides of the tube with the heel of my hand. Still no joy? At the point I usually consider the keyboard "too-far-gone" and get another.
Although I've used Macs since 1989, I have no experience setting up wireless devices on Windows. You may need to check with Windows support or a Windows forum to learn that if no one else offers a tutorial here.