Mac Mini w/ dual displays, primary and secondary

I have a Mac Mini with dual displays... Thunderbolt adaptor-VGA out and an HDMI out. I need the Thunderbolt-VGA display to be my primary. But each time I plug in the HDMI display, it becomes my primary. When I go into System Preferences -> Displays, then Arrangements, then Gather Windows... I see both displays, but the HDMI always appears as primary. Re-arranging does not seem to do any good. Reboots happen in the HDMI display, every time. Does anyone know how to change this, so that my Thunderbolt-VGA display becomes my primary?