10.9 VPN behind Airport Extreme no longer works

Same problem here. L2TP VPN setup was working fine on Mountain Lion 2 days ago. Yesterday, I upgrade to Mavericks, can no longer coonect to the server (PPTP works fine). Tried a clean install of Mavericks, fresh install of Server, still no luck. Reverted back to Mountain Lion and everything works as expected. I hope to see a fix for this soon.

Same problem as everyone else. Hopefully a fix will be out soon for this!

I spent 4 hours on the phone with Apple Enterprise Support (kept getting transferred up the support person chain). At one point we created a new account for the support person and they tried the VPN, which worked. At that point he told me if must be my connections on the other devices I was trying and that was all support could do- he also told me that when pinging my server he was seeing a lot of packet loss. After this I requested that he guide me through completely uninstalling the server, since every time I do this it carries over settings (right down to the shared secret) when reinstalling. He wanted to verify that we were doing it correctly so he put me on hold- at this point I ran a web based ping and traceroute (to rule out my local network) coming from Europe NO packet loss. We did a complete uninstall of the server portion and reinstall and the shared secret came back- which he could not explain. I then showed him the ping/traceroute- which he insisted since he could connect (which we could see in the logs) that it was my local networks for the other machines I was testing on, also that this was as far as support could go. For instance trying my iPhone on AT&T's LTE network and a work laptop- VPN into the work network THEN trying to come back to my VPN network. I informed him that I was going to completely nuke the machine and start over- he advised that I try the VPN from a Starbucks first, still insisting that it was my local cell/work networks being spotty.

I erased the main drive, re-installed Mavericks, then re-installed server 3 annnnnnnnd I am back to the same place. Currently I am testing from another location and still cannot get past the IKE Phase 1 portion of racoon's auth via hostname HOWEVER if I use Logmein to get back to another machine on the same network and use the IP I can use the VPN.

Here's the log:

Oct 23 08:22:10 hostname racoon[224]: Connecting.

Oct 23 08:22:10 hostname racoon[224]: IPSec Phase 1 started (Initiated by peer).

Oct 23 08:22:10 hostname racoon[224]: IKE Packet: receive success. (Responder, Main-Mode message 1).

Oct 23 08:22:10 hostname racoon[224]: >>>>> phase change status = Phase 1 started by us

Oct 23 08:22:10 hostname racoon[224]: IKE Packet: transmit success. (Responder, Main-Mode message 2).

Oct 23 08:22:10 hostname racoon[224]: IKE Packet: receive success. (Responder, Main-Mode message 3).

Oct 23 08:22:10 hostname racoon[224]: IKE Packet: transmit success. (Responder, Main-Mode message 4).

Oct 23 08:22:10 hostname racoon[224]: Connecting.

Oct 23 08:22:14 hostname racoon[224]: IKE Packet: transmit success. (Phase 1 Retransmit).

Oct 23 08:22:47 --- last message repeated 3 times ---

Oct 23 08:22:50 hostname racoon[224]: IKE Packet: transmit success. (Phase 1 Retransmit).

Oct 23 08:23:10 hostname racoon[224]: IKE Packet: transmit success. (Phase 1 Retransmit).

Oct 23 08:23:59 --- last message repeated 1 time ---

Oct 23 08:23:59 hostname racoon[224]: IKE Packet: transmit success. (Phase 1 Retransmit).

Oct 23 08:24:56 --- last message repeated 1 time ---

Oct 23 08:24:59 hostname racoon[224]: IKEv1 Phase 1: maximum retransmits. (Phase 1 Maximum Retransmits).

Oct 23 08:24:59 hostname racoon[224]: Phase 1 negotiation failed due to time up. 2194c11c97819d97:a29d73f04fe7e67f

The L2TP-VPN server did not respond. Try reconnecting. If the problem continues, verify your settings and contact your Administrator.

Here’s the network topology- Internet > Modem > Airport Extreme > Mac Mini via ethernet.

For the hostname DNS we are using a dynamic DNS service, which I have verified is resolving to the machine through the router ect.

I have tried deleting the Server App and /Library/Server as well as any pref files I could find, then rebooting, after downloading the Server App again I found all of my settings are back. Also I’ve tried removing the Server Setup Done file as well in conjunction as well as independently with no luck.

I have tried killing raccoon via the activity monitor as well as via the command line.

I am able to reach the machine locally via ssh and screen share, and externally via logmein.

I have tried an iPhone 5s locally and externally, and two MacBook Airs internally and externally as well.

Tried messing with racoon's access to private keys still no luck. Currently trying to restore a TM back in a VM to run on the machine.

I have deleted the VPN port forwarding entry in the Airport, tried putting it back manually as well as via the Server App and the drop down menu in the Airport.

I am 99% sure the traffic is reaching the server as I can see the following when I try to authenticate to the VPN, please note this is always the same for each VPN client

So at this point I am stuck either rolling back to ML or getting on the phone again with Enterprise Support who is convinced that it's not on them.

I erased the main drive, re-installed Mavericks, then re-installed server 3 annnnnnnnd I am back to the same place. Currently I am testing from another location and still cannot get past the IKE Phase 1 portion of racoon's auth via hostname HOWEVER if I use Logmein to get back to another machine on the same network and use the IP I can use the VPN.

I have another call with support tomorrow morning as I am starting to see message boards and App Store reviews saying the same thing- Server 3.0 seems to have broke the already fragile OS X VPN Server...

I guess it has something to do with ports on the firewall (just don't know which ones yet). If I try to connect to the VPN on my iPhone 5s when it is connected to the same LAN as the server, the VPN connection is set up correctly (defeats the purpose, but it works).

The firwall is definitely a component in the problem, but too many people with functioning Mountain Lion Server setups have had this same problem without any firewall changes. It's possible that Apple has made changes to their L2TP service that don't conform to the L2TP traffic specifications, which would explain why so many users with correctly configured firewalls are having the problem.

Mine was indeed working on Mountain Lion and stopped working on Mavericks. Going to try (risky for a production server) to forward all ports and see if that works. If so, then i'm going to try and find the culprit port by trial and error.

Did anyone get anywhere towards a solution here? I have the same problem; worked with 10.8.5 and Server 2.2 doesn't work in 10.9.0 and Server 3.0). If I try to have another domain name in VPN settings the connection dies immediately but if I have the same domain name in the VPN settings as the server has in it's own DNS the client is able to try to connect and is trying for a while and seems to be connecting to something but nothing shows up in the logs at the server.

When I think of it I might not have used the VPN with the last or maybe the two last updates of AirPort software. Could something be wrong there?

If you replace use/slib/racoon with the server 2.2 version it works again (be sure to reboot after replacing as the file is in use).

Sorry writing this on my ipad with autocorrection: it's /usr/sbin/racoon

Would you be so kind as to indicate how to obtain the raccoon from Server 2.2 and replace it on 3.0?

Yes, seems to work as you say. Did a back up copy of the file, copied the file from another server that runs Server 2.2 on 10.8.5 and rebooted. It works with a different domain name as well. Now I just got to test when not sitting on the same network and just going to the gateway and back in. Thanks for the help.

Thanks for the potential solution. Can you post the complete path for /usr/sbin/racoon? I'm not a total newb, but don't deal with OSX internal structure very often. Thanks,

That is the complete path.... /usr/sbin/racoon

It's a hidden directory, so you won't be able to navigate to it using Finder without using the menu option Go > Go To Folder..., or use Terminal.