You do not remember your encrypted backup password, I can understand.
You do not remember having created an encrypted backup, I can understand this also.
This means iCloud Key chain was checked while creating the encrypted backup, because by default it is checked.
Look for it in iCloud Keychain.
Did you un-check keychain while creating the encrypted password (By default it is checked) See the pic below.
If you had un-checked then the only option that you have is, keep trying various passwords as there is no limit on number of attempts.
You may have forgotten that you had entered a password. --> About encrypted backups on your iPhone, iPad, or iPod touch ...