iPhone X ear speaker not working during calls

I have had my iPhone X for a while. On occasion, my ear speaker stops working and I can only hear calls via the loud speaker. I think it has happened to me maybe 10 times in the last year and a half. I don't plug earbuds into the charger/earbud jack, only the charger. I always have to turn the phone off and on. It always gets fixed by doing this, but what can I do to fix it permanently?