7 Replies Latest reply: Mar 27, 2016 11:33 AM by Mrwhoopie1
DanIsaacson Level 1 Level 1

Can someone please confirm there is a conflict with the OS X Lion Server L2TP VPN and the Back to My Mac service.


I'm getting IKE packet errors on our VPN server and suspect someone on the network has Back to My Mac enabled.


Feb 14 12:12:12 server racoon[196]: IKE Packet: receive failed. (Responder, Quick-Mode Message 1).


"Authentication Failed" from the client side.  But after a few attempts it will connect... strange.

IPv4 IPv6 issues?


My router is non-Apple

and has the following NAT entries forwared to the Mac Server for the L2TP VPN


UDP 500

UDP 1701

UDP 4500


it also has UPNP enabled.


Does Back to My Mac use UPNP, IPv4, or IPv6 only?


Is Back to My Mac conflicting with the VPN service in Lion Server?

If so, WHY is this not documented?

Mac mini, Mac OS X (10.7.3), Lion Server (10.7.3)
  • Chris.Crisp Level 1 Level 1

    i'm pretty sure it's conflicting with Back to My Mac. i'd been fiddling around with it for days why i couldn't establish a VPN connection on a VPN gateway (win2k8r2) behind Airport Extreme (Back to My Mac) enabled.


    i thought at first it was the firewall, but it was not, then i checked the ports, i even tried PPTP, but i couldn't establish a successful connection even though i have the same VPN setup as with the one in our workplace.


    i then called Apple Support about this issue, i asked what ports Back to My Mac are using, he told me a dozen of ports which includes the ports for L2TP (1701, 4500, 500). and some in PPTP ports too i think, i can't remember them all.


    it would be nice though if the two (VPN and Back to My Mac) can run without conflicts. but yea... i knew already at first when i first used Back to My Mac it's basically a VPN linked to your Apple ID. simple/good implementation by Apple but i wish they used different ports right?


    edit: actually it is documented, it is written somewhere in Apple Support that you have to turn off Back to My Mac/MobileMe to use VPN. i just read it the other day.

  • DanIsaacson Level 1 Level 1

    So I've found the offical word burried in an Apple Document... 



    scroll down to Port 4500 which is used for IPSec VPN and Back to My Mac...


    4500UDPIKE NAT Traversal-ipsec-msftMac OS X Server VPN service, Back to My Mac (MobileMe, Mac OS X v10.5 or later).
    Note: VPN and MobileMe are mutually exclusive when configured through an Apple access point (such as an AirPort Base Station); MobileMe will take precedence.


    1) Do NOT create network services that conflict with well known and used TCP and UDP ports.

    2) If you create two conflicting services. Please MAKE A NOTE IN THE MANUAL for OS X Server so Admins are aware of the problem.

  • QuickTimeX Level 1 Level 1

    This is so disgusting. This issue wasted me hours of time.

    Is it not possible for Apple to pick different ports?

  • TheMomentum Level 1 Level 1

    This is still prevalent today on El Capitan. If anyone has OS X server installed, do not turn on Back To My Mac if you plan on having a VPN setup.


    Doing so knocked my server out, and i had to go through a significant amount of work to fix it.


    To fix:


    Turn off Back To My Mac, and do a kickstart


    sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/k ickstart -activate -configure -access -off -restart -agent -privs -all -allowAccessFor -allUsers              


    Then do a

    sudo launchctl stop com.apple.racoon

    sudo launchctl start com.apple.racoon


    and reboot, and when the server comes back up restart the VPN service through the OS X server console .

  • thavanam Level 1 Level 1



    I have tried your post, but still no luck.


    My VPN status shows Available - Reachability unknown


    I am able connect my vpn from another device on the same network. I am not able to connect from internet.

    I am getting the following error when I try to connect from my iPhone 4s.

    The L2TP-VPN server id not respond. Try reconnecting. If the problem continues, verify your settings and contact your administrator.


    OS - OS X El Capitan

    version 10.11.2


    Please help if any one knows the solution.

  • JesseDegenerate Level 1 Level 1

    wow, i'm on 10.10.5 for my server (10.11.2 for clients and iOS 9.2) and was pretty upset when i read your post, as i had just enabled it on a whim as i do a lot of remote stuff, but i figured i would play with it for some fun.


    I was able to get it going again rather quickly with both still active, my server showing up in back to mac, AND connecting via L2TP. There's a lot of information out there saying it's impossible, so i just figured i would say it's apparently not, so don't give up, although it may just have been fixed.


    I just re-forwarded the ports, toggled VPN. good to go.

  • Mrwhoopie1 Level 1 Level 1

    What did you do top make it work?Because i run all the latest software and since yesterday i activate "back to my Mac" and i can't connect via L2TP i'm not @ home right now and i remotely shut down "remote to my Mac" but still no VPN via L2TP