There is known problems with the 'insertion' switch being dicky on the 2010/2011 models of iMac, Mini and MBP (I've got the MBP 2011).
I've had this too. Cleaning it worked for a while. Well not cleaning it, but gently inserting a toothpick and 'stroking the left hand side, bottom' of the slot unsticks the switch which registers the 'insertion and removal' of the card.
But the reboot is strange...
Is anything showing up in the logs? Use spotlight to find 'Console', under system files, look for system.log and you should see when the card is inserted and removed (for the first time after boot).
If you can spot the entries, make a note of them in textedit or notes, so you know what they look like, remove the card, wait a minute or two and reinsert it.. Go back to the log and what do you see?
Another issue is that some card manufacturers, make the cards slightly too short.... I've had a few of these too!
I had to literally push them in, and pull them out a few mm until they were found.
Annoying yes, but this is why standards are made 😀
COuld you give that a shot and see how it goes?