iPhone volume comes through ear speaker and not bottom speaker

Sometimes when I get a call while I am taking a whatsapp call / watching a video with volume on, after i receive the call the volume only comes out of the ear speaker and not the main speaker. This happens multiple times and not necessarily always. I wish I could recreate the issue but it doesn't happen always. It works fine when i restart the phone.