I set up an Alpha-numeric passcode using the steps you indicated. What is now showing up is a regular keyboard like you see when you are typing. There is a 123 box on the keyboard. You would tap that to get the numeric display for that keyboard.
However, that type of keyboard only came up when I created a passcode that also had letters.
You can try to type in your regular 4-digit code, but I'm thinking that it wants something else at this point.
I was not familiar with that set of options in the passcode setup until you pointed them out to me, so I'm guessing that you explored the Passcode Options at some point? It sounds like maybe you did set up an Alpha-numeric passcode. It's not going to set one up for you - there is no logic on an iPhone that would be able to do that. Can you think back to when you were exploring the options and see if you can remember trying anything else? Can you try some combinations of letters and numbers that you may have used?
If you can't figure out what it is, then you won't have any other option except to erase the device and set it up again. Do you not have a backup of the device in iTunes or iCloud?
Cheers,
GB