Why do landscape videos end up in portrait in Photos?

I shoot lots of short videos in landscape mode with my iPhone 6. When I look at these videos in the Photos app, many (not all) show up sideways in portrait mode. If I rotate the phone to view the video properly in landscape mode, the video rotates to be sideways again in portrait mode, i.e., I'm holding the phone landscape, but the video plays in portrait, with big black bars on either side - and the video is still playing sideways. But some videos show up properly in landscape mode. Why does this happen? Does it have something to do with whether I have the phone's home button on the left side or the right side when I'm shooting in landscape mode?

These sideways videos show up the same wrong way in Photos on my Mac. I realize that I can export the videos, open them in Quicktime and rotate them there, but this is a major inconvenience, given that this happens so often. How can I prevent it?