Lifeproof Case and that ** Mute Button

Hi, I have my phone in a lifeproof case (well, to be fair, it's actually a knock-off, called a redpepper case) And when I occasionally remove the phone from said case, the mute button keeps getting switched by mistake. The access to it from the outside it broken, which means, to switch it back, I have to take the thing BACK out of the case, then put it back in. So, my question is this: would it damage the phone to stick that button with just a LITTLE bit of super glue?

(yes, I know, I should probably just get a REAL lifeproof case, but....)