Thunderbolt adapter keeps pulling out

I have a 27 Inch iMac with 2 monitors connected to it via thunderbolt.

One is connected via HDMI cable -> HDMI/thunderbolt adapter.

The other is connected via VGA cable -> VGA/thunderbolt adapter.

The one using VGA -> thunderbolt keeps pulling itself out. It happens about once a day. That monitor will go black, I'll look in back and, sure enough, the thunderbolt adapter has pulled part way out of the port, enough that it loses the connection.

Has anybody else experienced this? Does anybody have a "life-hack" or fix of some kind?

Thanks in advance!