Hey there, Clubmoss!
Thanks for stopping by the Apple Support Communities. After reading your post, we understand that your iPhone 12 mini will not rotate into landscape mode while using the Messages app, though it does work in other apps. You're in the right place for help!
It looks like you have already made sure the Orientation Lock is not enabled, which is a great first step. Since it is still not working, the next thing we recommend is making sure Display Zoom is not enabled. You can do this by going to Settings > Display and Brightness > Display Zoom. If the toggle is on, turn it off. Then, open Messages and see if landscape works.
Should you continue to have trouble after that, don't hesitate to respond back!
Have a lovely rest of your day.