Mac mini with dual thunderbolt to VGA not waking second monitor and randomly rebooting

So I have a Late 2014 mac mini (OS X 10.10.4 I keep it up to date; 3 GHz Intel Core i7; 16gb ram; 256 factory solid state) that is having sleep issues. For my monitors I have 2 dell E193FP (older 19inch flat panel monitors with VGA input only) both are connected via Thunderbolt->VGA (real apple store bought ones) adapters. I have also tried to move one to a HDMI->VGA adapter and got the same results.

The issue I am having is that when the mac mini goes to sleep it will do 1 of 3 different things.

1. During night sleep(me and the mac are supposed to be sleeping) I hear a bong and look over to see that mac had decided to reboot itself. It will then reboot to the screen that says it can't find a mouse or keyboard. It will keep the screen awake indefinitely until I come type on the keyboard to wake the keyboard up. I can then login and use as normal. The setup is in my room so this is especially annoying when it happens at 3am for no reason. It also freaks my wife out. It also does this during the day sometimes, but most of the day it is awake and I am working on it.

2. I will try to wake it up and only the main screen will come on. I can do anything in the OS I want to try to wake the second screen, but it will not be recognized by the mini till I unplug the TB adapter and plug it back in. It is worth noting that when I unplug it they main screen will go black for a second and when I gather windows it gathers the settings window for the second screen. It is like it knows it is there, but forgot to wake it.

3. I try to wake it up and everything is prefect all is well. VERY RARE, but it does happen

What I have tried :

Detect Screens when it screws up

Disabled Powernap

Disabled autopoweroff in pmset

Anyone have any ideas or know what to do?