Read the server logs for the current error, concentrating on Apache. That'll usually tell you what isn't working.
Check the Apache configuration file syntax:
apachectl configtest
Verify DNS with the command:
sudo changeip -checkhostname
If you're on a NAT'd network and running local DNS services, stay out of .local and .arpa with the domain you're using, and preferably use a registered domain. If you're NAT'd, don't try to use ISP or Google or other off-network DNS servers, as those do not provide translations for the private-IP-block NAT'd IP addresses you're using.
As for a reset, there's a server setup done file (I don't recall the filename of that off-hand), but I've found removing that file does not reset the entire environment completely and correctly — my usual preferred approach is to set up the new 10.6 server environment, get DNS configured and operating first and only then enable the other necessary services (proper DNS is vital to a stable environment, and incorrect DNS is the trigger for more than a few weird problems with OS X Server), and then migrate over the web files from a backup. That's the most effort, but generates the "cleanest" environment.
Less preferred, but can work: using an external backup of the old OS X Server environment and migrating that data over during the installation of 10.6 is another option, but that can sometimes migrate in the old problems.