Voiemail Call Failed

I was using Google Voice for voicemail with my old android phone. When my wife upgraded her iPhone and I took her old one, I continued using the google voice service.

I recently decided to not use it anymore, and have disconnected my number from it. But now when someone calls me and it goes to voicemail, they get "Could not complete your call, please try again". I was told to reset my network settings, which I did, but it is still doing the same thing. When I go to the voicemail tab in the Phone app, it looks normal and does not show that it needs to be set up.

Any advice?