While you can disable Siri (the speech assistant), you currently can’t disable the Siri button (also used for general Search). You can give feedback to Apple, asking for such a setting. Feedback - Apple TV - Apple
I don’t share your experience, though. I find the remote easy to pick up without pressing the side button, even when placing a finger over the button. It’s only 64 grams (2.2 ounces), it doesn’t need grabbing like a hammer. And there is enough tactile feedback to distinguish between touching and pressing the button, similar to the other buttons.