Is it the first generation SE?
When there is a sudden drop like this, my first though is maybe you should do a battery calibration?!
In a case like this, I suspect the battery not even being that bad, but just giving out a false reading.
I had an iPhone 6s sitting on 1% for hours (percentage jumping around and dropping in large amounts instantly like in your example) and powering on multiple times after it has shut itself off due to the 'empty' battery which it shouldn't be able to if the battery reading would be accurate. After doing the following procedure, it worked a whole lot better. It still has the original battery with 84% health. Granted it is the 6s and really rather old already, and while it didn't improve overall battery life by much, doing the calibration fixed the erratic percentage drops and made the device a little bit more reliable because 'what you see is what you get' and not a plummeting battery meter that is lingering on 1% for hours.
To calibrate: (it is best do to this on the weekend when you are less likely do not have any place to be early in the day etc.)
1) Drain your device until it shuts off. Reboot or at least try to until it doesn't do so anymore (also do it by the hard reboot method, as sometimes the device boots whereas it wouldn't boot if only the power button is held).
2) Leave unplugged for a couple hours (overnight is best)
3) charge the device until full and then leave plugged in for at least two more hours.
4) unplug, then do the whole procedure one more time
Be vigilant about those steps as only performing them the right way, will actually give you any real and good result. Good Luck!