I've had this problem since December 2016 on my 3 month old Macbook Pro 13inch Early 2015.
Guys, this issue is nothing to do with FileVault. FileVault just expose this problem to you. This is a faulty hardware issue.
I have two macbook pro. exactly same setup. One is showing this issue. another is perfectly fine.
Problem is "EVERYTHING" before OS load up is slow. you can try it. Hold Alt to go to boot device selector, you will see it is slow. Hold D when Macbook is turning on, you will see Diagnosis page is also slow. FileVault login page is a pre-boot screen. It ask you for password before it can unlock your hard disk. once password is entered, Real boot process start and OS load up.
I am not sure why it is fast inside of OS, but before OS loads up, it is painfully slow with keyboard and mouse lagging.
I had sent my Macbook to repair on 26 December, got it back on 3rd Jan. They did a SMC reset, NVRAM reset and a full OS clean install and they told me my Macbook Pro is flawless!
Issue went away temporarily but returned back as soon as I used SD Card slot. I am not sure if your issue is also related to SD Card slot usage, but it could indicate some hardware parts inside your Macbook is faulty.
Since now I figure out it is returning back when I use a SD Card, I returned it back to repair again this Monday and told them to try putting an SD Card. Now they find out there is a hardware error and they are fixing the parts. I shall receive my Macbook back next week and I will update here on what they did change and If issue went away.
P.S. This issue might look like FileVault issue because it is part of a pre-boot screen. since all pre-boot screens are slow, you will see it first thing. I suspect there are lot of users with this issue, but they are unaware of it because they don't use FileVault. However boot is going to be slow still, apple logo will take 30 to 60 seconds to show up.
Contacting Apple Care at this moment, and reporting about this problem by telling them filevault login screen is slow, won't guarantee you a fix because they are clueless about issue. You need to figure out exact source of problem on your Macbook first, and then tell the issue to technician.