I'm a pretty determined guy and have spent the past few days trying to resolve this. I have also spent several hours on the phone with Apple Enterprise Support (kept getting transferred around). At one point we created a new account for the support person and they tried the VPN, which worked.A number of things were tried to no avail, We did a complete uninstall of the server portion and reinstall and the shared secret came back- which he could not explain.He insisted the problem was with my network or the devices I was using to try and connect. I informed him that I was going to completely nuke the machine and start over- he advised that I try the VPN from a Starbucks first, still insisting that it was my local cell/work networks being spotty. I know this is not the case.
I erased the main drive, re-installed Mavericks, then re-installed server 3 annnnnnnnd I am back to the same place. Currently I am testing from another location and still cannot get past the IKE Phase 1 portion of racoon's auth via hostname.
Here's the log:
System Log
Oct 23 23:48:36 imac.local racoon[238]: IPSec Phase 1 started (Initiated by peer).
Oct 23 23:48:36 imac.local racoon[238]: IKE Packet: receive success. (Responder, Main-Mode message 1).
Oct 23 23:48:36 imac.local racoon[238]: >>>>> phase change status = Phase 1 started by us
Oct 23 23:48:36 imac.local racoon[238]: IKE Packet: transmit success. (Responder, Main-Mode message 2).
Oct 23 23:48:36 imac.local racoon[238]: IKE Packet: receive success. (Responder, Main-Mode message 3).
Oct 23 23:48:36 imac.local racoon[238]: IKE Packet: transmit success. (Responder, Main-Mode message 4).
Oct 23 23:48:39 imac.local racoon[238]: IKE Packet: transmit success. (Phase 1 Retransmit).
Oct 23 23:49:12 --- last message repeated 4 times ---
Oct 23 23:49:12 imac.local racoon[238]: IKE Packet: transmit success. (Phase 1 Retransmit).
VPN Service Log
2013-10-23 23:47:23 CDTLoading plugin /System/Library/Extensions/L2TP.ppp
2013-10-23 23:47:23 CDTListening for connections...
On each Device I tried from various locations I would get the message ....
The L2TP-VPN server did not respond. Try reconnecting. If the problem continues, verify your settings and contact your Administrator.
Here’s the network topology- Internet > Modem > Airport Extreme > iMac via ethernet.
I have a Static IP provided by Comcast BusinessClass, and have verified that all traffic routes are working properly.
I have tried deleting the Server App and /Library/Server as well as any pref files I could find, then rebooting, after downloading the Server App again I found all of my settings are back. Also I’ve tried removing the Server Setup Done file as well in conjunction as well as independently with no luck.
I have tried killing raccoon via the activity monitor as well as via the command line.
I am able to reach the machine locally and via ssh and screen share, and externally via logmein, Parallels Access and ARD.
I have tried an iPhone 5s, 4s, iPad2, iPad 3rd gen all with iOS7 locally and externally, and two MacBook Airs internally and externally as well both with Maverics.
I have deleted the VPN port forwarding entry in the Airport, tried putting it back manually as well as via the Server App and the drop down menu in the Airport.
I am sure the traffic is reaching the server as I can see the system log entries when I try to authenticate to the VPN, please note this is always the same for each VPN client.
So at this point I am stuck either rolling back to ML or getting on the phone again with Enterprise Support who is convinced that it's not on them.