Skip navigation

set up vpn on mac osx server

6808 Views 13 Replies Latest reply: Feb 14, 2014 6:31 AM by FredrikHedman RSS
johnsenisi Calculating status...
Currently Being Moderated
Aug 8, 2013 4:08 PM

What is the most common reason that I can connect to the newly created vpn locally but not remotely (over the wan)?

OS X Mountain Lion (10.8.4)
  • bfdulock Level 1 Level 1 (135 points)
    Currently Being Moderated
    Aug 8, 2013 5:00 PM (in response to johnsenisi)

    The most likely reason is a misconfiguration or non-configuration of the router/firewall.  By that I mean the proper ports are not open or forwarded correctly.


    Assuming you are using an OS X Server, the following table shows what ports should be open/forwarded on your router for VPN to work from outside the local network:





    UDP 500    -->  internal server address

    UDP 1701  -->  internal server address

    UDP 4500  -->  internal server address


    IP-ESP (IP protocol 50, ESP)     passthru enabled






    TCP 1723    -->  internal server address


    IP-GRE (IP protocol 47)     passthru enabled




    *** See for further information:




    Bryan Dulock

    Houston, TX

  • bfdulock Level 1 Level 1 (135 points)
    Currently Being Moderated
    Aug 10, 2013 8:24 AM (in response to bfdulock)

    Another reason could be an incorrect IP address.



    Bryan Dulock

    Houston, TX

  • bfdulock Level 1 Level 1 (135 points)
    Currently Being Moderated
    Aug 10, 2013 10:39 AM (in response to johnsenisi)



    - when connecting over the WAN, is your computer on an outside network?

    - when connecting over the WAN, are you using the public IP of your router?

    - is the AirPort Base Station the actual router?

    - what does the VPN log on the server say?

  • bfdulock Level 1 Level 1 (135 points)
    Currently Being Moderated
    Aug 16, 2013 5:13 AM (in response to johnsenisi)

    Is the Time Warner modem configured in bridge mode?  If not, the TW modem is acting as the router and the AirPort Extreme would be superfluous as a router.  This would also explain why VPN from the outside is not working.

  • bfdulock Level 1 Level 1 (135 points)
    Currently Being Moderated
    Aug 16, 2013 5:31 AM (in response to johnsenisi)

    Another possibility, it the TW modem is acting as your router, is the put the AirPort Extreme in bridge mode and just use the TW modem as your router.  Sometimes that is a simpler setup.

  • bfdulock Level 1 Level 1 (135 points)
    Currently Being Moderated
    Aug 16, 2013 5:50 AM (in response to bfdulock)

    Sorry about the typos above.  Here's a rewrite:


    Another possibility, assuming the TW modem is already acting as a router, is to leave it as the router and put the AirPort Extreme in bridge mode.  Sometimes this is a simpler setup.

  • Oakleef Level 1 Level 1 (0 points)
    Currently Being Moderated
    Oct 28, 2013 4:42 AM (in response to johnsenisi)



    I'm having similar problems, but with the difference that I can't access VPN over LAN nor WAN. If I change the address that I'm connecting to, to which ofc is my internal IP, it works, but when I type my external IP address or it doesn't.


    For portforwarding I've got 500, 1701, 4500 for UDP and 1723 for TCP.


    Any ideas? I'm drawing blanks here. Tried almost everything... Except the thing that makes it work.


    My websitedomain is forwarded to my IP from my host and that works just fine, but VPN won't...

  • techboss Calculating status...
    Currently Being Moderated
    Oct 30, 2013 8:46 PM (in response to johnsenisi)

    Same problems with L2TP server on Mavericks after upgrade.


    I turned on logging level to in /etc/racoon.conf:


    # "log" specifies logging level.  It is followed by either "notify", "debug"

    # or "debug2".

    log debug2;


    Restarted vpnd and tried to connect and then checked out my console.  I saw the connection attempt being made and all looked ok up until it got stuck hitting numerous:


    10/30/13 11:28:13.544 PM racoon[348]: Malformed cookie received or the spi expired.


    errors then finally logging:


    10/30/13 11:30:08.964 PM racoon[348]: Resend Phase 1 packet e9d548d778586159:6eae3dbe3fe45f70

    10/30/13 11:30:41.961 PM racoon[348]: IKEv1 Phase 1: maximum retransmits. (Phase 1 Maximum Retransmits).

    10/30/13 11:30:41.961 PM racoon[348]: Phase 1 negotiation failed due to time up. e9d548d778586159:6eae3dbe3fe45f70

    10/30/13 11:30:41.961 PM racoon[348]: Disconnecting. (Connection tried to negotiate for, 169.760629 seconds). failure Phase 1 negotiation failed (Maximum retransmits). NAT detected by Me 169.760629



    10/30/13 11:30:41.961 PM racoon[348]: IKE Phase 1 Failure-Rate Statistic. (Failure-Rate = 100.000). noop IKE Phase 1 Failure-Rate Statistic 100.000



    10/30/13 11:30:41.961 PM racoon[348]: Freeing IKE-Session to[60405].

    10/30/13 11:30:41.961 PM racoon[348]: IV freed



    Out of gas for the night trying to track this down...anyone else care to try to move the discussion forward?

  • FredrikHedman Calculating status...
    Currently Being Moderated
    Feb 14, 2014 6:31 AM (in response to techboss)

    I have had the same problem with setting up L2TP on Mavericks after upgrade.  After several failed attempts I have the following recepie.


    To test this you need to have two separate networks to connect you VPN client to.  One should be the same as where the server is running and the other needs to be different so that the incoming traffic to your router is coming from the outside.


    I'm assuming a setup with a router and behind it a local network with an OS X server running the VPN service (vpnd daemon)


    On the server

    • Note the local ip-adress of your server.  This should preferably be static.
    • Install the VPN fix from apple:
    • In the OS X Server VPN Service create a VPN profile where VPN Host Name is local ip-adress of the VPN server.
    • Restart the VPN service and save the configuration file.


    On the router

    • Open ports 500, 1701 and 4500 to pass UDP traffic to the server.  Make sure to activate them in the router interface.
    • Make a note of your routers public IP address. This should be static.
    • If this keeps changing you can set up a dynamic domain name (
    • Optional: verify that the ports are actually open using nmap:


    sudo nmap -Pn -sU XX.XX.XX.XX -p500,1701,4500
    Starting Nmap 6.40 ( ) at 2014-02-14 14:21 CET
    Nmap scan report for ... (XX.XX.XX.XX)
    Host is up (0.012s latency).
    PORT     STATE         SERVICE
    500/udp  open          isakmp
    1701/udp open|filtered L2TP
    4500/udp open|filtered nat-t-ike
    Nmap done: 1 IP address (1 host up) scanned in 1.29 seconds


    • XX.XX.XX.XX is the public IP-adress of the router.  You can also try the same on the local IP-address of the server.



    On the client

    • Copy the configuration file and install it by double klicking on the file.
    • Connect the client to the same local network as the vpn-server and activate the VPN connection. 
      • Verify that the VPN connection comes up.
      • Up to this point, smooth sailing. 
    • Now change the Server address to the IP-address of the router and turn on extra logging found under Advanced. Save the new configuration.
    • Bring up the VPN connection again.  Should work.  Right?
      • I did not for me.  The error complains about the L2TP-VPN-server not responding.
      • Digging deeper using the system logger I found the error
    2014-02-14 14:43:31,039 racoon[60284]: IKE Packet: receive failed. (Malformed or unexpected cookie).
    2014-02-14 14:43:31,039 racoon[60284]: Malformed cookie received or the initiator's cookies collide.
    2014-02-14 14:43:31,172 pppd[60283]: IPSec connection failed
    2014-02-14 14:43:31,172 racoon[60284]: vpn_control socket closed by peer.
    2014-02-14 14:43:31,173 racoon[60284]: received disconnect all command
      • So it sort of works, but complains about some bad cookie.
      • The simple change of the IP-address apparentely generates this error.
    • Now change the network of the client so that it is not on the same networks as the server.
    • Bring up the VPN again.  Now it just works.
    • So apparently, when the traffic is coming in from the outside the VPN connection just works.
      • If you change back to the local network of the server and the keep the router IP-address the error is back.


    • The conclusion is that the client used for connecting to the VPN network must be on an outside network.
    • In retrospect, this makes sense since we should test using an environment that reproduces the actual use case. The crux is to ensure that the client traffic is coming in from the outside.


    Hope this helps.



    OS X Mavericks (10.9.1), OS X Server, VPN Mavericks


More Like This

  • Retrieving data ...

Bookmarked By (3)


  • This solved my question - 10 points
  • This helped me - 5 points
This site contains user submitted content, comments and opinions and is for informational purposes only. Apple disclaims any and all liability for the acts, omissions and conduct of any third parties in connection with or related to your use of the site. All postings and use of the content on this site are subject to the Apple Support Communities Terms of Use.