Remote Server Administration

I have OS X server installed on a Mac Mini in my home office. I was using TeamViewer to log into the computer to administer the server until I read that I could install OS X Server on other devices and perform any setting changes, etc. from there.

However, when putting in my host name, administrator name, and password I am unable to access the server. When I connect to the VPN that is through that server I am able to connect to the server. Any other connections I am not able to connect.

The "Using server app on Remote Mac" is checked.

I am just learning the whole server world and any help would be appreciated.

Thanks!