iPhone doesn’t download my voicemails

Often I will receive a phone call, and I will suspect that the the caller left a voicemail, but my phone app says I have no voicemails. If I call my own phone number to check my voicemails, sure enough I will have a new message. So my iPhone isn’t downloading the message for some reason. Sometimes, days after the phone call, the voicemail will randomly download. Neither restarting my phone, nor resetting the network settings have helped.