iPhone X stuck on passcode screen with OK button

Somehow my FaceID stopped working and also when I tried entering the passcode, it won't let me login even though I am entering the right passcode. On a closer look, I noticed that the passcode was way tinier than normal and as soon as I enter something, it shows an "OK" button next to the passcode field. Never seen this before.

Has anyone seen this before and any idea on how to fix this? Apple support over phone was not much of help. They suggested to restore the device.

iOS version : 11.2.6