That may be the only downside, but it's not much different than using your thumb for the digital crown.
When I first saw the arrangement of buttons its asymmetry struck me as being glaringly un-Apple. I thought it would be more consistent with Apple's design philosophy if the crown were placed in the middle, with the button in the middle of the opposite side. That way, left-handed users wearing it on the right wrist (for example) would enjoy exactly the same appearance and functions as right-handed users, but reversed. The orientation with the crown opposite the wrist now makes me think they devoted more thought to the design than I had originally assumed.
Scrolling through messages (for example) with my left thumb tends to shield the right side of the Watch from onlookers also.
It's an odd orientation only if users are for having a crown in its "traditional" place but leave it to Apple to break tradition. There is no logical reason it has to be there. After using it even for a short time I think the digital crown makes a lot more sense on the elbow side of the Watch... at least for me.