Forgot my FileVault pass and is stuck at file with question mark

ok so I had a adware called Pirrit which messed up my Mac pretty badly

so I decide to factory reset my Mac hut after that it made a different FileVault pass which made my Mac unable to install an OS

which now its stuck on the file with an question mark and when I press command + R

it gives me a bar where I type

i really dont want to go to the Apple Store to fix it

is there any other way to get my Mac back to normal?