The most likely reason is a misconfiguration or non-configuration of the router/firewall. By that I mean the proper ports are not open or forwarded correctly.
Assuming you are using an OS X Server, the following table shows what ports should be open/forwarded on your router for VPN to work from outside the local network:
FOR VPN USING L2TP over IPSec
UDP 500 --> internal server address
UDP 1701 --> internal server address
UDP 4500 --> internal server address
IP-ESP (IP protocol 50, ESP) passthru enabled
FOR VPN USING PPTP
TCP 1723 --> internal server address
IP-GRE (IP protocol 47) passthru enabled
*** See for further information:
I checked all settings, everything is correct per Apple's directions. The connection is allowed on the LAN. There is no WAN connection. VPN is turned on so the tunnel is available. Airport Base Station has the correct ports enabled (OSX Server did this for me in the set up process). I am at a loss as to why a connection outside the LAN is not possible.
I bought the new cable modem, the Time Warner leased modem had no functionality (and there is a monthly cost).
I set up Apple Base Station to bridge mode.
Internet functionality is back with new Zoom cable modem dishing out the ip addresses, etc.
However, now the only way I can connect on the LAN is by typing the explicit ip addess of the server in the VPN configuration screen. Using the VPN Host Name will not connect.
I don't even dare trying the WAN until I get the LAn working correctly.
I'm having similar problems, but with the difference that I can't access VPN over LAN nor WAN. If I change the address that I'm connecting to, to 10.0.1.21 which ofc is my internal IP, it works, but when I type my external IP address or vpn.mydomain.com it doesn't.
For portforwarding I've got 500, 1701, 4500 for UDP and 1723 for TCP.
Any ideas? I'm drawing blanks here. Tried almost everything... Except the thing that makes it work.
My websitedomain is forwarded to my IP from my host and that works just fine, but VPN won't...
Same problems with L2TP server on Mavericks after upgrade.
I turned on logging level to in /etc/racoon.conf:
# "log" specifies logging level. It is followed by either "notify", "debug"
# or "debug2".
Restarted vpnd and tried to connect and then checked out my console. I saw the connection attempt being made and all looked ok up until it got stuck hitting numerous:
10/30/13 11:28:13.544 PM racoon: Malformed cookie received or the spi expired.
errors then finally logging:
10/30/13 11:30:08.964 PM racoon: Resend Phase 1 packet e9d548d778586159:6eae3dbe3fe45f70
10/30/13 11:30:41.961 PM racoon: IKEv1 Phase 1: maximum retransmits. (Phase 1 Maximum Retransmits).
10/30/13 11:30:41.961 PM racoon: Phase 1 negotiation failed due to time up. e9d548d778586159:6eae3dbe3fe45f70
10/30/13 11:30:41.961 PM racoon: Disconnecting. (Connection tried to negotiate for, 169.760629 seconds).
com.apple.message.signature: Phase 1 negotiation failed (Maximum retransmits). NAT detected by Me
10/30/13 11:30:41.961 PM racoon: IKE Phase 1 Failure-Rate Statistic. (Failure-Rate = 100.000).
com.apple.message.signature: IKE Phase 1 Failure-Rate Statistic
10/30/13 11:30:41.961 PM racoon: Freeing IKE-Session to 184.108.40.206.
10/30/13 11:30:41.961 PM racoon: IV freed
Out of gas for the night trying to track this down...anyone else care to try to move the discussion forward?
I have had the same problem with setting up L2TP on Mavericks after upgrade. After several failed attempts I have the following recepie.
To test this you need to have two separate networks to connect you VPN client to. One should be the same as where the server is running and the other needs to be different so that the incoming traffic to your router is coming from the outside.
I'm assuming a setup with a router and behind it a local network with an OS X server running the VPN service (vpnd daemon)
On the server
- Note the local ip-adress of your server. This should preferably be static.
- Install the VPN fix from apple: http://support.apple.com/kb/DL1716
- In the OS X Server VPN Service create a VPN profile where VPN Host Name is local ip-adress of the VPN server.
- Restart the VPN service and save the configuration file.
On the router
- Open ports 500, 1701 and 4500 to pass UDP traffic to the server. Make sure to activate them in the router interface.
- Make a note of your routers public IP address. This should be static.
- If this keeps changing you can set up a dynamic domain name (http://dyndns.org)
- Optional: verify that the ports are actually open using nmap:
sudo nmap -Pn -sU XX.XX.XX.XX -p500,1701,4500 Password: Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-14 14:21 CET Nmap scan report for ... (XX.XX.XX.XX) Host is up (0.012s latency). PORT STATE SERVICE 500/udp open isakmp 1701/udp open|filtered L2TP 4500/udp open|filtered nat-t-ike Nmap done: 1 IP address (1 host up) scanned in 1.29 seconds
- XX.XX.XX.XX is the public IP-adress of the router. You can also try the same on the local IP-address of the server.
On the client
- Copy the configuration file and install it by double klicking on the file.
- Connect the client to the same local network as the vpn-server and activate the VPN connection.
- Verify that the VPN connection comes up.
- Up to this point, smooth sailing.
- Verify that the VPN connection comes up.
- Now change the Server address to the IP-address of the router and turn on extra logging found under Advanced. Save the new configuration.
- Bring up the VPN connection again. Should work. Right?
- I did not for me. The error complains about the L2TP-VPN-server not responding.
- Digging deeper using the system logger I found the error
2014-02-14 14:43:31,039 racoon: IKE Packet: receive failed. (Malformed or unexpected cookie). 2014-02-14 14:43:31,039 racoon: Malformed cookie received or the initiator's cookies collide. 2014-02-14 14:43:31,172 pppd: IPSec connection failed 2014-02-14 14:43:31,172 racoon: vpn_control socket closed by peer. 2014-02-14 14:43:31,173 racoon: received disconnect all command
- So it sort of works, but complains about some bad cookie.
- The simple change of the IP-address apparentely generates this error.
- Now change the network of the client so that it is not on the same networks as the server.
- Bring up the VPN again. Now it just works.
- So apparently, when the traffic is coming in from the outside the VPN connection just works.
- If you change back to the local network of the server and the keep the router IP-address the error is back.
- The conclusion is that the client used for connecting to the VPN network must be on an outside network.
- In retrospect, this makes sense since we should test using an environment that reproduces the actual use case. The crux is to ensure that the client traffic is coming in from the outside.
Hope this helps.