That should not be the case; you may want to make a Genius Bar appointment to have your phone examined.
Don't worry; if it is a hardware issue, that's why there is a factory warranty.
You can verify how audio is supposed to work on any of the demo iPhone XS/XS Max models on display in the store.