Hi,
Server 5 uses a reverse proxy (front end) which stands in front of various http-based network services, which I will refer to as backend services. In this configuration, a single apache config is used to handle all of the incoming requests from other hosts - apache_serviceproxy.conf. Incoming requests are reverse proxied to back end services, based on the request URI and whether a given back end service is enabled or disabled.
To create a custom vhost, click the + button under the websites list in Server.app. Define the vhost the same as in previous releases; using some unique combination of domain name, IP address, and port. When you save the site, a new config file will be written into /Library/Server/Web/Config/apache2/sites. The vhost you just created won't be served by an apache instance that's listening on ports reachable from other hosts, but that's OK because the reverse proxy will be ready to proxy those requests to your new vhost, based on configuration that Server adds to /Library/Server/Web/Config/Proxy/apache_serviceproxy_customsites.conf.
One possible reason that your previously created vhosts aren't working might be that those configs are not meant to be behind a reverse proxy. In theory, custom sites that you had created in previous versions of Server should have been converted at upgrade time, but any hand-written / edited configs might not be handled automatically. In this case, you may need to edit those files to look similar to a custom site created by Server 5, or just re-create them from scratch using the Server UI.
I would not recommend trying to 'undo' the reverse proxy setup in Server 5. Hopefully I've explained how you can still add your own custom sites in a way that integrates with the new Server 5 reverse proxy setup. Also, take extreme caution when editing any files that you didn't create. Doing things in the apache config that aren't supported by the UI often causes issues down the road, because the upgrade / migration scripts can't anticipate all the possibilities of a hand-edited config.
Hope this helps.
Cheers,
-dre
