Apple’s Worldwide Developers Conference returns June 10, 2024

Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

OS X Server / VPN /The L2TP-VPN server did not respond...HELP!

I am very new to OS X Server and my goal is to setup DNS & VPN! I would like to have this setup to be able to connect into my apple computer from work or friends house. I am using an Apple Airport Extreme router and im also using the latest version OS X Mountain Lion with OS X Server installed. I have started an account with dyndns website for user host name (using a _____@dyndns.org address). I assume this would be used as an alternate way of being able to connect without starting a personal website. I also signed up for another site (no-ip) and I now have a different IP address (not sure if that was necessary). I then followed instructions on youtube (instructional videos by todd for OS X Server Mountain Lion) which seemed to be very easy to understand. But after setting up my VPN on the client side (network setting in system preferences), i tried to connect VPN (L2TP) and i receive this error message "The L2TP-VPN server did not respond. Try reconnecting. If the problem continues, verify your settings and contact your Administrator.". When I open Consol in the utilities folder, I am seeing part of the following message below;


racoon[117]: IKE Packet: transmit success. (Phase1 Retransmit).

racoon[117]: IKE Packet: receive failed. (malformed or unexpected cookie).

pppd[490]: IPSec connection failed

Does anyone know what's happening or what I need to do to fix this? Or can someone tell me the basic requirements to setting things up correctly?

OS X Mountain Lion (10.8.2)

Posted on Mar 6, 2013 9:03 PM

Reply
Question marked as Best reply

Posted on Mar 7, 2013 2:03 AM

Two things you didn't mention that I suggest you try:


1. at your firewall (router, modem, whatever), you need to open the ports used by the VPN protocol you are using. When the remote computer tries to connect to your server, your router has to pass the info from these port(s) on o your server so your server sees the incoming communication. Youll need to forward any incoming info on those particular ports on to your server's INTERNAL IP address (ex., 192.168.1.xx). Also, you'd want to set your server's internet prefs to have a static IP address inside your network. That way, it doesn't grab a randomly assigned IP address every time it boots up. If your server's internal IP address changes often, then your router can't forward traffic to it.


2. The address of your server will be <server>.dyndns.org, not <server>@dyndns.org. Note the dot, not the @.

19 replies
Question marked as Best reply

Mar 7, 2013 2:03 AM in response to Sparty28

Two things you didn't mention that I suggest you try:


1. at your firewall (router, modem, whatever), you need to open the ports used by the VPN protocol you are using. When the remote computer tries to connect to your server, your router has to pass the info from these port(s) on o your server so your server sees the incoming communication. Youll need to forward any incoming info on those particular ports on to your server's INTERNAL IP address (ex., 192.168.1.xx). Also, you'd want to set your server's internet prefs to have a static IP address inside your network. That way, it doesn't grab a randomly assigned IP address every time it boots up. If your server's internal IP address changes often, then your router can't forward traffic to it.


2. The address of your server will be <server>.dyndns.org, not <server>@dyndns.org. Note the dot, not the @.

Mar 7, 2013 8:18 PM in response to cpragman

Ok...I started completely over tonight by reinstalling Mountain Lion and OS X Server. I have an Airport Extreme so ports used for VPN should automatically transfer to the router. I am using the server address name "server.example.private" and i made sure to change my shared secret name.


When i try to connect to my VPN, I am getting the same error "The L2TP-VPN server did not respond. Try reconnecting. If the problem continues, verify your settings and contact your Administrator."

The only strange thing i dont understand is when i click on my Airport Extreme, i see that i have my IP address "98.255.XX.XX" and i have a LAN IP address "10.0.1.1". Should i use the 98.255.XX.XX for setup in OS X Server or 10.0.1.1? This has been really frustrating for me....is there any hope on getting this to work?


Note: I was told that its not necessary to have a dyndns or no-ip IP account when setting up a VPN. So I am not using any of these services currently.


Any help would be appreciated....

Mar 8, 2013 4:31 PM in response to Sparty28

Your particular error message suggests that packets aren't reaching your server. You'd get a different error message if the server refused the conneciton due to password mismatch.


Need more info:

What appliance do you have providing your internet to your home/office? The thing upstream of your airport base station. Can you describe your network topology, starting at the cable modem/DSL router?


I suspect that you haven't opened the VPN ports on your cable modem/DSL router. That is the device that needs to forward VPN packets from the outside world to your server. You'll need to go thru whatever configuration interface that device has.


When you are outside your home/office, you'd connect to your server by connecting to the 98.255.xxx.xxx address. Dyndns and no-ip make this part easy by generating a URL that always points to your home/office router, even if your ISP periodically re-assigns IP addresses.


10.0.1.1 is your "private" IP address inside your network. It is meaningless outside your network. The airport base station takes one signal in, and generates "private" IP addresses for all the devices inside your network. Traffic that leaves your network and goes out onto the public internet has this private address removed by the router, and replaced with your "public" IP address (provided by your ISP). This is how all the devices in your home/office can share a single IP connection provided by your ISP. It's called Network Address Translation (NAT). It's a little too technical, but this might help -> http://en.wikipedia.org/wiki/Network_address_translation


If both your Cable Modem/DSL Modem and your Airport Base Station are providing NAT, then that can lead to complications. It's best to turn one or the other off. On my network, I've got the DSL modem doing NAT, and beneath that an Airport Extreme that I've set to "Bridge mode" so it doesn't do "Double NAT". That also means only on device to configure as far as opening VPN ports. This explains Double NAT -> http://support.iprimus.com.au/index.php?Itemid=214&id=517&option=com_content&tas k=view

Mar 9, 2013 10:35 AM in response to cpragman

Im using Comcast for my ISP and from the wall I have a Motorola Surfboard 6120 cable modem (not sure how to access my setting on the modem). So basically I have my 6120 cable modem connected to the Apple AirportExtreme router and is then wirelessly connected to my macbook pro. im providing screen shots of my apple router settings, OS X Server settings and firewall (which is turned off) settings. Any suggestion on how i should set things up or if you can tell me step by step would be greatly appreciated.


User uploaded file

User uploaded file

User uploaded file

User uploaded file

User uploaded file

User uploaded file

Mar 9, 2013 11:57 AM in response to Sparty28

Couple of things.


On your 3rd screenshot, you have VPN turned on. That's good, but doesn't show me what port you are forwarding the VPN traffic to. Make sure that that item is forwarding VPN traffic to the INTERNAL IP address of your server (ex., 10.0.1.2).


On your 4th screenshot, you've listed your own computer as being the source of DNS info. You need to replace that with the IP address of a DNS server OUT THERE in the internet. I've attached a screenshot of what I use (Open DNS servers). Your computer will provide DNS to other computers (and itself) that are within your network, but to GET that info for itself, it must be looking to a DNS server outside your network to get it's info from.


On the second screenshot, remove the Comcast DNS entry (75.75.76.76), and just put your own server's DNS in twice (10.0.1.2). There's no precedence as to which of these servers one of your computers will try, so when they try the comcast DNS server, they won't see the features that are being advertised as available by your own server. By lacing your server in both blanks, all macs that join your network will be forced to use your server as the DNS server. Your server will then get it's own DNS info from the outside DNS server you manually enter in the server config.


User uploaded file


User uploaded file

Mar 10, 2013 5:33 AM in response to Sparty28

The screenshots show your server's ACTUAL private IP address is 10.0.1.4. You have configured all your other settings as if your server's address is 10.0.1.2. They need to match.


I system prefs on your server, open the network settings, and change it from automatic to manual and manually specify it to have IP address 10.0.1.2.

Mar 10, 2013 10:41 AM in response to Sparty28

So in your third screenshot above, on your client Mac. Instead of entering "server.whatever.private" in the Server Address box, you'll need to enter the public IP address of your network (98.255.whatever.whatever).


If you have a DynDns account pointing to your server, then you would use that in the address field instead of a hard-coded IP address number. That's optional at this point, but helpful in the future.

Mar 10, 2013 3:21 PM in response to cpragman

See the picture for where to enter your public IP address. This is a configuration you make on the laptop/portable, not on the server. You're telling the laptop the IP address (e.g., phone number) it needs to know to call the server.


You would enter the IP address that your ISP assigns your router here (98.255.something.something).


User uploaded file


Now if you have registered your server with DynDNS, No-Ip, etc., you'd have a URL name you set up with them (ex., joeblowserver.dyndns.com). You could enter THAT URL in this box shown above instead. That way, if/when your ISP changes your public IP address (which they do occassionally), you could still access your server.


I forgot to mention that testing VPN requires that you take your laptop OUTSIDE of your own network, to see if it works. You'd have to go to a friend's house, coffee shop, etc. If you're trying to VPN to your server from inside your own network, it probably won't work.

Mar 10, 2013 3:57 PM in response to cpragman

Oh...ok...im kind of confused. So I should be able to connect into my home network with my iPhone 5 correct? If I connect using my Verizon LTE I should have no issues getting into my computer? Do need to setup my proxy for the iPhone?


Also for the VPN, I thought you should be able to setup your VPN on OS X Server to create a more secure connection to the outside Internet (world wide web). So if I was downloading data from from the Internet or searching on the web or signing into personal accounts online, all Internet traffic data would be tunneled or encrypted...correct?

Mar 10, 2013 4:43 PM in response to Sparty28

Kind of right, kind of wrong.


Yes, you can use the VPN feature on your phone to tunnel all data traffic (encrypted) to your server at home via VPN. It would then go out from your server to the internet, in a standard format. That's good for doing stuff on your phone that your are worried about being eavesdropped on by someone sniffing wireless packets in a coffeeshop or some other non-protected wireless situation. The VPN feature on the iPhone works both over Wifi and cellular data (LTE, 3G, etc.). You need to configure the VPN on your phone one time (access via settings). After that, you'd just flip the on/off switch you see in the settings to turn it on/off.


Your server is your "home base" that your phone or laptop connect to via the encrypted VPN tunnel. The data would then go out onto the internet from your server to it's ultimate destination on the WWW. That part of the journey would not be encrypted by VPN. It would just be standard WWW data packages from that point on.

OS X Server / VPN /The L2TP-VPN server did not respond...HELP!

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.