If the password for the encrypted backup is not recognized, you'll have to start all over again:
Warning: If you encrypt an iPhone backup in iTunes and then forget your password, you will not be able to restore from backup and your data will be unrecoverable. If you forget the password, you can continue to back up and use the device, however you will not be able to restore the encrypted backup to any device without the password. You do not need to enter the password for your backup each time you back up or sync.
If you cannot remember the password and want to start again, you must perform a full software restore and when iTunes prompts you to select the backup from which to restore, choose set up as a new device.
copied from iTunes: About iOS backups