With AT&T: Open your iPhone to dial a call. Press and hold the number 1 until a voice answers (listen carefully, the voice was very soft for me). It gives a number of options, including resetting your password or turning off the password for voicemail. I turned off my password for voicemail.
The standard password for AT&T voicemail is the 10-digit phone number that the phone is registered with, but that did not work for me. I used the steps above.