Visual voicemail not available after every call

Every time I end a call on my iPhone 5, visual voicemail is unavailble for about 10 seconds. It appears that my cell connection is not present during this time as I cannot access Siri during this time as well (If I'm outside of WIFI). I know it isn't hardware related as I had to swap ouf my phone because of a different issue and the same behavior continues. Has anyone else seen this and for a fix for it?