I bought my Apple TV box in 2015.
You specified your relevant device as “Apple TV HD, tvOS 17”. That model [A1625] was first sold in 2015, although there was some overlap with the prior model. (The 3rd generation wasn't discontinued until 2016.)
It should be compatible with the current latest version tvOS 18.5.
About Apple TV 4K and Apple TV HD software updates - Apple Support
How to update your Apple TV - Apple Support
What happens when you try to update? Do get an error massage? What does it say exactly?
It won't update to the newest iOS version.
How odd. tvOS 17.x to tvOS 18.x isn’t that big of a jump in versions. Try contacting Apple Support about this. They may suggest that you arrange for a visit to an Apple Store or an Apple Authorized Service Provider. They may alternatively suggest to try to restore & update the system software through a computer, via USB. That latter method would erase existing apps and preferences though.
I purchased a movie that won't play on the older version.
Again, how odd. iTunes Store movies shouldn’t have a dependency on Apple TV box system software version.
What happens when you try to play this movie? Do get an error massage? What does it say exactly?
As of tvOS 17.2, iTunes movies and TV shows have moved to the Apple TV app (TV app). But the message in the ‘old’ Movies app explains that. The old apps are now obsolete. The Apple TV app would have your familiar Library content.
Do I have to purchase a new one?
I don’t think so; not for this reason alone.