Q: screen size and font enlarge at random.

Hello themeyers1,

Thanks for using the Apple Support Communities!

I understand that the display on your Mac zooms in randomly and you are not sure why. So far the only way you have found to resolve this is to restart your Mac. If this is correct, I do have a couple of suggestions for some settings to look at. 

I know you are using a Mac mini, but not sure whether you use a Magic Trackpad or Magic Mouse with it. It could be possible that you are inadvertently activating Smart zoom while using your Magic Trackpad or Magic Mouse. The following two articles explain how this feature works and where to find it in System Preferences if you wish to adjust the setting. 

Use Multi-Touch gestures on your Mac - Apple Support

OS X El Capitan: Use an Apple Magic Mouse

If the Smart Zoom is not the culprit, it is possible that Zoom may be enabled in Accessibility with either a keyboard combination or mouse/keyboard combination that is inadvertently zooming in your display. 

The following two articles explain where the Zoom settings are in Accessibility and how it works. 

OS X Yosemite: Zoom pane of Accessibility preferences

OS X Yosemite: Make the screen image larger using an Apple mouse

Cheers!