Had similar problem (could VPN to Mountain Lion Server 2.2.2 with my Mavericks Macbook Pro. When trying to VPN to the server from my IOS 7.0.4 iPhone, I could get a connection, but wasn't able to access my Contacts on the server - verification kept failing with and without SSL).
I had created the VPN profile using the Server app and then saved it and emailed it to my IOS 7.0.4 iphone. I also had access to an IOS 5.x iPad, and installed the same profile there. Contacts sync worked fine on the IOS 5 device.
I finally deleted the VPN profile on the iphone, and manually created one on the iPhone 7.0.4 by typing in the account info from the Server User pane and from the Server VPN pane.
It was a bit long typing in the shared secret, but when I was finished, I was able to connect, and now I could - on the iPhone - create a new Contacts account including server information, and the verification was successful. I can now access the Contacts on the iPhone and on my server.
Am guessing something in IOS 7.0.4 misreads the VPN profile, which causes problems down the line. Apple?
Good luck bgw8probe!
Have you tried removing special characters from your Shared Secret? I was having iOS problems with 10.9.4 and server 3.1.2 (it never worked in 10.9 with server 3), and now everything's connecting!