Apple TV Siri remote battery

I am having an issue with my Apple TV Siri remote. It's been working fine until a couple of weeks ago. It stopped working and I assumed it needed a charge. I plugged it up for several hours and then unplugged it to use but still no luck. I plugged it back up for a couple more hours but it still does not work. It shows that it's charging when using my iPhone to navigate to the remote section of the Apple TV. This is where it gets really weird. I then figured out that the remote works perfectly while it's plugged up and charging but as soon as I unplug it, it quits working. It doesn't matter if it charges all day, it only works if I keep it plugged up. It seems the battery will not hold a charge. Any ideas other than buying a new remote or continuing to use my iPhone?