set up vpn on mac osx server

The most likely reason is a misconfiguration or non-configuration of the router/firewall. By that I mean the proper ports are not open or forwarded correctly.

Assuming you are using an OS X Server, the following table shows what ports should be open/forwarded on your router for VPN to work from outside the local network:

FOR VPN USING L2TP over IPSec

--------------------------------------------------------------------------

UDP 500 --> internal server address

UDP 1701 --> internal server address

UDP 4500 --> internal server address

IP-ESP (IP protocol 50, ESP) passthru enabled

FOR VPN USING PPTP

--------------------------------------------------------------------------

TCP 1723 --> internal server address

IP-GRE (IP protocol 47) passthru enabled

*** See for further information:

http://support.apple.com/kb/TS1629

Bryan Dulock

Houston, TX

Another reason could be an incorrect IP address.

Bryan Dulock

Houston, TX

Questions:

- when connecting over the WAN, is your computer on an outside network?

- when connecting over the WAN, are you using the public IP of your router?

- is the AirPort Base Station the actual router?

- what does the VPN log on the server say?

Is the Time Warner modem configured in bridge mode? If not, the TW modem is acting as the router and the AirPort Extreme would be superfluous as a router. This would also explain why VPN from the outside is not working.

Another possibility, it the TW modem is acting as your router, is the put the AirPort Extreme in bridge mode and just use the TW modem as your router. Sometimes that is a simpler setup.

Sorry about the typos above. Here's a rewrite:

Another possibility, assuming the TW modem is already acting as a router, is to leave it as the router and put the AirPort Extreme in bridge mode. Sometimes this is a simpler setup.

Hi.

I'm having similar problems, but with the difference that I can't access VPN over LAN nor WAN. If I change the address that I'm connecting to, to 10.0.1.21 which ofc is my internal IP, it works, but when I type my external IP address or vpn.mydomain.com it doesn't.

For portforwarding I've got 500, 1701, 4500 for UDP and 1723 for TCP.

Any ideas? I'm drawing blanks here. Tried almost everything... Except the thing that makes it work. 🙂

My websitedomain is forwarded to my IP from my host and that works just fine, but VPN won't...

Same problems with L2TP server on Mavericks after upgrade.

I turned on logging level to in /etc/racoon.conf:

# "log" specifies logging level. It is followed by either "notify", "debug"

# or "debug2".

log debug2;

Restarted vpnd and tried to connect and then checked out my console. I saw the connection attempt being made and all looked ok up until it got stuck hitting numerous:

10/30/13 11:28:13.544 PM racoon[348]: Malformed cookie received or the spi expired.

errors then finally logging:

10/30/13 11:30:08.964 PM racoon[348]: Resend Phase 1 packet e9d548d778586159:6eae3dbe3fe45f70

10/30/13 11:30:41.961 PM racoon[348]: IKEv1 Phase 1: maximum retransmits. (Phase 1 Maximum Retransmits).

10/30/13 11:30:41.961 PM racoon[348]: Phase 1 negotiation failed due to time up. e9d548d778586159:6eae3dbe3fe45f70

10/30/13 11:30:41.961 PM racoon[348]: Disconnecting. (Connection tried to negotiate for, 169.760629 seconds).

com.apple.message.domain: com.apple.Networking.ipsec.connect.plain

com.apple.message.result: failure

com.apple.message.signature: Phase 1 negotiation failed (Maximum retransmits). NAT detected by Me

com.apple.message.value2: 169.760629

10/30/13 11:30:41.961 PM racoon[348]: IKE Phase 1 Failure-Rate Statistic. (Failure-Rate = 100.000).

com.apple.message.domain: com.apple.Networking.ipsec.phasestats.plain

com.apple.message.result: noop

com.apple.message.signature: IKE Phase 1 Failure-Rate Statistic

com.apple.message.value: 100.000

10/30/13 11:30:41.961 PM racoon[348]: Freeing IKE-Session to 166.147.119.217[60405].

10/30/13 11:30:41.961 PM racoon[348]: IV freed

Out of gas for the night trying to track this down...anyone else care to try to move the discussion forward?

I have had the same problem with setting up L2TP on Mavericks after upgrade. After several failed attempts I have the following recepie.

To test this you need to have two separate networks to connect you VPN client to. One should be the same as where the server is running and the other needs to be different so that the incoming traffic to your router is coming from the outside.

I'm assuming a setup with a router and behind it a local network with an OS X server running the VPN service (vpnd daemon)

On the server

• Note the local ip-adress of your server. This should preferably be static.
• Install the VPN fix from apple: http://support.apple.com/kb/DL1716
• In the OS X Server VPN Service create a VPN profile where VPN Host Name is local ip-adress of the VPN server.
• Restart the VPN service and save the configuration file.

On the router

• Open ports 500, 1701 and 4500 to pass UDP traffic to the server. Make sure to activate them in the router interface.
• Make a note of your routers public IP address. This should be static.
• If this keeps changing you can set up a dynamic domain name (http://dyndns.org)
• Optional: verify that the ports are actually open using nmap:

sudo nmap -Pn -sU XX.XX.XX.XX -p500,1701,4500
Password:

Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-14 14:21 CET
Nmap scan report for ... (XX.XX.XX.XX)
Host is up (0.012s latency).
PORT     STATE         SERVICE
500/udp  open          isakmp
1701/udp open|filtered L2TP
4500/udp open|filtered nat-t-ike

Nmap done: 1 IP address (1 host up) scanned in 1.29 seconds

XX.XX.XX.XX is the public IP-adress of the router. You can also try the same on the local IP-address of the server.

On the client

• Copy the configuration file and install it by double klicking on the file.
• Connect the client to the same local network as the vpn-server and activate the VPN connection.
  • Verify that the VPN connection comes up.
  • Up to this point, smooth sailing.
• Now change the Server address to the IP-address of the router and turn on extra logging found under Advanced. Save the new configuration.
• Bring up the VPN connection again. Should work. Right?
  • I did not for me. The error complains about the L2TP-VPN-server not responding.
  • Digging deeper using the system logger I found the error

2014-02-14 14:43:31,039 racoon[60284]: IKE Packet: receive failed. (Malformed or unexpected cookie).
2014-02-14 14:43:31,039 racoon[60284]: Malformed cookie received or the initiator's cookies collide.
2014-02-14 14:43:31,172 pppd[60283]: IPSec connection failed
2014-02-14 14:43:31,172 racoon[60284]: vpn_control socket closed by peer.
2014-02-14 14:43:31,173 racoon[60284]: received disconnect all command

• So it sort of works, but complains about some bad cookie.
• The simple change of the IP-address apparentely generates this error.
• Now change the network of the client so that it is not on the same networks as the server.
• Bring up the VPN again. Now it just works.🙂
• So apparently, when the traffic is coming in from the outside the VPN connection just works.

  If you change back to the local network of the server and the keep the router IP-address the error is back.

Conclusion

• The conclusion is that the client used for connecting to the VPN network must be on an outside network.
• In retrospect, this makes sense since we should test using an environment that reproduces the actual use case. The crux is to ensure that the client traffic is coming in from the outside.

Hope this helps.