Ok having more success and then some extra wierdness - but first the success:
I did have the DNS stuff set up wrong - I saw haykong posted a setup procedure for when you have another server for DNS but since that was not the case for me (and possibly others here) I'll post my run through.
I'll say that I was able to do this with an already set up server but I'm sure starting from scratch would be safer. Also my server has a static IP address (192.168.1.2).
1. In server app in the server section I set my hostname to something in this form "servername.companyname.local". I then set my computer name to servershortname.local (server app actually did this for me when I clicked the new repair button within the alert it threw about the computer name not matching the hostname)
2. In the certificates section I set Secure services to the new certificate made by the above step and then deleted all the other certificates (the show all certificates option was checked true under the gear menu)
3. Set up DNS serving
• Set the forwarding servers to whatever is your current DNS (for me it was my router 192.168.1.1 but I could easily see networks where it would just be what your ISP provides you)
• under the gear menu make sure "Show all records is true"
• I then deleted all the records (including the zones - yes you can select/edit/delete the section headers)
• I then added the primary zone with the format "companyname.local" (matching before) with all the setting left as defaults
• I then added the machine record with hostname "servername" (matching before) and added ip address of my server to the list. The zone was the one I had just created.
• Once clicking done on the machine record it created all the other records needed (looked like Primary zone: machine, nameserver | Reverse zone: reverse mapping, nameserver)
• Then I turned DNS on
4. At this point I had to nuke my open directory server and remake it - yes it stinks but I couldn't figure out a way for directory server to start using the new locations any other way
After remaking the Open Directory Server all the crashing when adding new users and changing passwords went away and I could log into those user from my clients.
Note for client setup: your network account server will not show up until you change the DNS of your clients to the IP address of your server. So for me I switched from 192.168.1.1 (my router/old DNS host) to 192.168.1.2(my server IP). Also sometimes I had to add the network account server twice for it to work right - I could tell when it was right when it only threw one warning will connecting instead of two)
...but here comes the weirdness. Some of the accounts when I go to log into it says the files are in the wrong position. It'll let you log in but none of your files will show up - if I navigate the computer another folder will appear titled "Networ" (yes missing a k) that will contain my home folder mount but the actual "Network" folder is empty. On network account that load normally the "Network" folder will be like normal and there will be no "Networ" folder.
So at this point at least I'm able to connect to the open directory remotely consistently just sometimes the paths are returned messed up.
Any thought?