9 Replies Latest reply: Nov 8, 2013 3:55 PM by TLightfoot
John Lockwood Level 5 Level 5 (5,360 points)

Apple have (and still do) use the open-source Racoon software as the behind the scenes code to provide their OS X Server VPN service. As accessed via the Server.app frontend this can be configured to do L2TP or PPTP connections, both of which are compatible with Macs and iPhones.


However it appears Racoon can also do Cisco IPSec style VPN connections. I have seen an article here http://blog.dest-unreach.be/2011/03/03/iphone-compatible-ipsec-vpn-on-an-ubuntu- server-with-ldap-authentication discussing doing this with Racoon on Ubuntu.


Now the reason for the interest in this is that the VPN on Demand feature supported by the iPhone and iPad only works (currently) with Cisco IPSec or SSL VPN connections. So my hope is that the following may be possible.


  • Racoon can do Cisco IPSec (hopefully also under OS X)
  • Racoon can hopefully also do certificate authentications (Snow Leopard Server supported this even if Mountain Lion does not)
  • The above combination therefore in theory should work for VPN on Demand for iOS devices


Presumably if I use Server.app to turn on the VPN service it is going to wipeout any manual settings of Racoon?

  • 1. Re: Custom VPN Server config using Racoon?
    John Lockwood Level 5 Level 5 (5,360 points)

    Unfortunately it turns out that Apple have modified the standard Racoon. Apple's version apparently only supports authentication to system accounts via the passwd file, however OS X does not use that method for accounts, let alone the fact I want to use Open Directory accounts. Apple appear to have put their authentication code in vpnd which only does L2TP and PPTP and not Cisco IPSec.


    I have logged an enhancement request with Apple for them to add support for IPSec in OS X Server since that (or SSL VPN) is required for implementing VPN on Demand. Since the original version of the code Apple use can do this it would merely be a matter of including the missing modules when they compile it.


    I did explore trying to compile the standard IPSec-Tools/Racoon under OS X, unfortunately it failed with too many errors for me to be interested in fixing. As an aside I was disappointed to see even Apple's own IPSec-Tool implementation available to download here http://www.opensource.apple.com/source/ipsec/ fails to compile successfully in XCode 4.6 under OS X 10.8.2 although with far fewer errors.


    In the meantime I have been able to -


    • Install VirtualBox on OS X
    • Install Ubuntu Server in VirtualBox
    • Install IPSec-Tools and Racoon with LDAP support
    • Set it up to use a pre-shared key and allow client to authenticate to Open Directory via LDAP


    The Linux VM running Racoon now successfully accepts IPSec connections from Mac and iOS clients and authenticates them against Open Directory. All required routing and NAT is also working. All traffic from the clients is (as desired) forced to go via the VPN connection, clients can successfully access LAN and Internet hosts and DNS lookups work. So far exactly what I want Apple to do natively in OS X.


    Next step is to change from PSK to Certificate based authentication using a self-signed certificate.


    Step after that will be to buy a trusted certificate.


    I will then be able to support VPN on demand using just a Mac (sort of ) and at least with a self-signed cert having spent no money

  • 2. Re: Custom VPN Server config using Racoon?
    TLightfoot Level 1 Level 1 (0 points)

    Did you follow the guide at dest-unreach.be?  I tried following that guide on Ubuntu Desktop but with local system accounts instead of LDAP.  I can successfully connect to the VPN, but I cannot communicate with any of the LAN IP's.  What do I need to do to get all client traffic successfully working through the VPN gateway?  Also could it be that I used Ubuntu Desktop?  Maybe the server version is required?  I chose the Desktop version to make it as easy as possible for others to figure out how to add new system accounts.  Also I tried adding a DNS4 entry in my config file and when that is added, I cannot connect to the VPN.  If I comment it out, I can again connect to the VPN, but can't get to LAN IP's.  I have the config file set to assign IP's from the same subnet as the office LAN I am trying to VPN into.


    Thank you for your help,

    - Taylor

  • 3. Re: Custom VPN Server config using Racoon?
    John Lockwood Level 5 Level 5 (5,360 points)

    Yep I was following that guide but as I mentioned in my own reply Apple have modified their version of Racoon and it is not possible with the Apple version. It did work in Ubuntu. (I used the server version.)


    I am not where the server is right now but I will try and get access and post some notes here for you, so keep an eye out.


    I was using two Ethernet connections, one on the DMZ to the outside world to receive VPN connections, and one on the LAN to allow the remote users to reach the LAN via the VPN server. A lot of people would these days frown on this and only have the VPN server with one connection and use port-forwarding to reach it.


    I have not yet got round to adding the use of SSL certificates instead of a pre-shared key, next on my list.


    This would then result in a Cisco IPSec (compatible) VPN server, routing all traffic, and using certificates and hence in theory be suitable for 'VPN on Demand' on iOS devices.

  • 4. Re: Custom VPN Server config using Racoon?
    TLightfoot Level 1 Level 1 (0 points)

    This is essentially what I am trying to get working, but behind a sonicwall router.  I have set up a virtual machine in my offices Xenserver cluster and this virtual machine has a private LAN IP.  I have set up the firewall/nat in our sonicwall router to forward UDP 500 and 4500 as well as ESP 50 to the virtual machine running Ubuntu Desktop that I am trying to get working as a VPN gateway.  The LAN network is  The virtual machine is and I have set the VPN clients to receive addresses starting at up through  I'm guessing there's something I need to do to the Ubuntu machine to get the VPN clients communicating with other LAN IP's, but I don't know what needs to be done to get that working.


    Thank you for your help.

  • 5. Re: Custom VPN Server config using Racoon?
    John Lockwood Level 5 Level 5 (5,360 points)

    Apart from the fact I am using two physical network interfaces for my Ubuntu VM and thus avoiding having to do port forwarding I have the same setup. I even have a SonicWALL as the firewall.


    The VPN clients will be in their own private IP range - not part of the LAN range, they need to be in a different subnet As a result you need to configure the SonicWALL which I presume like mine is acting as the LAN default gateway to have a static route so it knows that to send traffic back to the VPN users the traffic has be routed via the Ubuntu servers LAN IP address. Otherwise any replies from the LAN intended for the VPN users will end up being sent out to the Internet and 'lost'.


    I am still digging out the config files I used but hopefully the above has helped.


    As a related topic I also have it authenticating the users via LDAP to Open Directory.

  • 6. Re: Custom VPN Server config using Racoon?
    TLightfoot Level 1 Level 1 (0 points)

    I believe I have the Ubuntu VM correctly setup now.  I have it set to assign VPN clients IP's that are -


    Here's part of my config file.


    mode_cfg {

    auth_source system; # Authenticate against Unix user database

    save_passwd on; # Allow users to save passwords


    network4;  # Give clients addresses starting from this address

    pool_size 50;  # up to 50 addresses higher

    # netmask4

    # dns4



    I commented out the netmask and dns entry because enabling them caused the client to fail to connect to the VPN gateway.



    The Ubuntu VM's IP is

    The Sonicwall is the office's main router and DHCP server.

    The main office network is interface X0: V10 on the Sonicwall, We have a few networks all on X0 and we use VLAN's on that interface to trunk them over into our network switch.

    The modem for our office is interface X1


    Can you give me some hints as to what I have to configure in the Sonicwall to get the VPN clients the ability to access the Internet through the office modem as well as the ability to talk to IP's on the network?

  • 7. Re: Custom VPN Server config using Racoon?
    John Lockwood Level 5 Level 5 (5,360 points)

    You need to configure an address object in the SonicWall for the 10.1.11.x network range. You then need to define a static route in the SonicWall to the the 10.1.11.x network via the address. This is because the 10.1.11.x network is not directly connected to the SonicWall so it needs to know how to 'reach' it.


    The VPN clients should be able to send traffic to the LAN via the VPN server, and with the above static route LAN devices will be able to send traffic to VPN clients, including replies. Without this LAN devices including the LAN servers will send all traffic not for the 10.1.10.x range only to the SonicWall and it will not know where to reach the 10.1.11.x range.


    A first test would be to have a VPN client ping a device on the LAN, then to try pinging an Internet address e.g. (Google DNS server).

  • 8. Re: Custom VPN Server config using Racoon?
    TLightfoot Level 1 Level 1 (0 points)

    Okay, I added the VPN IP Pool as an item and added the static route you described and I can now access devices on the office LAN via their IP address.


    The next issue is that I cannot access the Internet from the VPN client.  (I set up the VPN gateway to have the VPN client send all traffic over the VPN)


    The other issue is that we have some office intranet websites running off of the same internal IP.  So without the VPN client using our Office DNS server ( the VPN client can only access the default web site by going to the web server's local IP.  Office computers use DNS names to access the various different web sites being hosted on the same machine.


    I tried adding a DNS4 entry to the config file so that the VPN clients use the DNS server that is on the office LAN, but when I add the DNS4 entry and restart Racoon, VPN clients are unable to connect.


    I'm guessing that if I can get the VPN clients to use the DNS server, I'll be able to access all of the intranet websites via their non IP domain names and I should be able to also access the Internet since our internal DNS server forwards unknown requests to (Google DNS)


    Any tips on these last couple issues?


    Thanks a lot, you've been very helpful!

  • 9. Re: Custom VPN Server config using Racoon?
    TLightfoot Level 1 Level 1 (0 points)

    I found out why it wasn't working with my DNS4 and Netmask4 entries; I forgot to put semicolons after each line entry.  All is working as expected now.  Thank you!