Sorry to hear that. When I finally got a replacement unit, I note that it was pretty fast at first. The "multitasking" switcher for app selection was stuttery, and that's definitely just an underpowered hardware issue - both CPU and GPU IIRC are about twice as fast, but at 4K vs HD it's pushing 4x the number of pixels - general scrolling and menu selection was fine, though and Bluetooth was just as responsive as the HD unit. It felt more or less just as fast as the old HD unit.
BUT - after a few hours that suddenly it all went south - the remote disconnected & reconnected and then, well, that was that. The UI started to struggle & the remote wasn't reliable anymore. Brand new unit, same old problem.
My guess is that something along these lines happens: The bluetooth chip / board in the 4K breaks down, perhaps from hot/cold cycles, or just warming up for the first time. Bad solder joint, bad board connection, whatever. The failure mode might be generating a large number of spurious hardware interrupts so the software system is running hard to deal with them all. That's why the whole device seems to slow down once the Bluetooth connection issue arises.
I know from my own tests and the repair cycles that this is a hardware fault. It is not fixed by software resets or remote changes, and either stays the same or worsens over time. The phone app works because it's going over WiFi not Bluetooth, but even then the UI will continue to visibly lag and struggle because the faulty bluetooth unit inside the Apple TV is still malfunctioning. At least the 4K still seems to play video OK - just the UI seems to start to fail - and yes, you can use the phone as a remote effectively. Going back to the older HD was a relief in the sense that the UI and remote works flawlessly, but you lose out on HDR and ARC doesn't work so well (perhaps because of fewer supported audio formats, and/or because it's only got HDMI 1.x support, not HDMI 2).
You've probably had a broken unit for long enough that it misbehaves even when first powered on "at the wall" and cold, so if it's under warranty, it'll be easy to demonstrate the issue and I would suggest you seek a replacement. There's a good chance it'll also fail, but if so, you can eventually aim for a refund under warranty if it can't be fixed, maybe?
Just as an alternative, bear in mind that the 2020 Chromecast (https://store.google.com/us/product/chromecast_google_tv) looks really amazing, performs extremely well, is only $49.99 USD and you can install all kinds of fun stuff like retro games console emulators. Yes, it won't (of course!) run the Apple TV app so iTunes content can't be played in 4K, but there's such a big price difference that you could just re-purchase quite a few movies off the Play Store and still have money left over.
I don't understand Apple's strategy with any of this - their offerings are so poor that they're kinda forcing me into the opposing ecosystem :-(