Depending on the exact set-up of the network, L2TP requires UDP ports 500, 1701 and 4500 and the IP-ESP protocol, which is IP protocol 50; ESP.
Other than ESP (which is protocol 50 and not port 50), these are UDP ports, and not TCP.
TCP 1723 is used for PPTP. Not L2TP.
It is common for L2TP passthrough to fail when more than one connection is active.
As compared with L2TP, PPTP is usually easier to get going.
Check your Mac OS X Server firewall, too.
Use of an external firewall-gateway with an embedded VPN server is recommended. (NAT passthrough is something best avoided.)
Also ensure your ISP is not blocking VPN connections. There are ISPs that block server-oriented ports on the residential service tier. (If you're on a business-class tier, ignore this.)
Thank you for your reply. Sorry, I forgot to say that some of the ports I forwarded were UDP. My server firewall is not the problem because when I try to connect, I don't see any "blocked" messages in the log.
I forwarded UDP 500, 1701, and 4500. I forwarded TCP 1723 because I was also going to try to get PPTP to work, but it didn't. I'm also forwarding GRE and ESP.
My setup seems to be within the requirements needed for VPN to work except that I don't know if my ISP allows VPN connections. I didn't know they were even allowed to block that because their statement says that they allow any traffic that is lawful. There is some talk about them blocking 443, 25, GRE, and inbound 80, but those are on old forums. They MIGHT actually be blocking GRE, 443, and 25 because I also can't host my mail service. My site works on port 80 though.
FWIW, "seems" is a comparatively hazardous choice when debugging networking access, and determining permissible access. Test your access, and definitely confirm what is permissible and not within your ISP terms of service, or check directly with your ISP and get some email on what is and is not allowed.
If it's permissible within your ISP service tier, probe the specific ports using telnet or the openssl s_client and (particularly for this case nc (netcat) tools, and see if the ports allow access. nc can run port probes on UDP, which is the key piece here given those other two tools target TCP and TCP SSL connections. Probably something like the nc -zu w.x.y.z udp-port command.
Or alternatively, reconfigure your firewall back to disallowing VPN passthrough, try the VPN, and see if you can provoke the firewall to log hits on the ports.
If you know that your SMTP port 25 is blocked (your phrasing around the cause of the problems isn't clear to me), then you're on some analog to a DHCP-based residential service tier, and those can have other port blocks.
If your mail is messed up and port 25 is not blocked, then it's likely your DNS that's messed up, as more than a few folks have tried setting up SMTP without having their forward and reverse DNS working for the IP address of their mail server. For SMTP to work, your forward DNS and your reverse DNS and your MX record must all match. If this is the usual setup, the dig -x w.x.y.z of your external IP address will not match, and that'll often cause remote SMTP servers to choose to avoid your server.
Mail hosting and VPN work fine behind the router. Of course, I can't actually use the mail host for anything useful. I can't find anything on Verizon's site that says that they block any ports. In fact, I read the agreement, and they said that they allow any legal traffic. I'm going to ask the customer support.
I have an ActionTec router that came with FiOS. It lets me use pretty advanced settings, but the interface is annoying. It had presets for L2TP port forwarding, and they were the same as the ones recommended at http://support.apple.com/kb/TS1629?viewlocale=en_US
That shows the ports typically used by Mac OS X Server.
And I don't have MobileMe. I'm self-hosting
Please use a remote client to probe the specific ports using telnet or the openssl s_client tool or - and particularly for this case - the nc (netcat) tools, and see if the ports allow access. nc can run port probes on UDP, which is the key piece here given those other two tools target TCP and TCP SSL connections.
Probably something like the nc -zu w.x.y.z udp-port command, where your public static IP address is w.x.y.z.
Confirm that your ports are open and reachable, in other words.
The solution by @MrHoffman is what put me on the right track to a solution.
For me (under the 7.6 firmware on my TimeCapsule) with port-mapping UDP ports 1701,4500,500 to the internal private IP addresss (which was fixed to always be the same thanks to configuring DHCP Reservations based on MAC addresses) turning VPN on with L2TP on my iPAD failed when outside my network... even though it succeeded when inside (at home) my network. Watching the port activity on the sever (internal) using 'sudo tcpdump -i en0 port 500 or port 1701 or port 4500' showed that no packets were making it to the internal machine. The Log of iVPN was also silent during this failure. Although I told airport to send level 5 syslogs to the internal machine,.. nothing showed up. ARGHH.
Replacing the L2TP port-mappings with PP2P (beurk) port-mappings (TCP:1723) SOLVED MY PROBLEM.
So thanks, Mr.H,... you're my hero.
I am in the same/similar boat as you guys. I'm on Verizon FIOS, and I just installed a new Airport Extreme. I took my sever off the internet directly, and put it behind the router (formerly it WAS the router, too).
Here's the funny thing: I set up something very similar for a client, and they can connect no problem. That is, I can connect, right now, no problem. They are on an older AEBS, but it's also running 7.6 firmware. I set up matching port mapping rules, 500, 1701, 4500 all UDP, and 1701 TCP. Neither server has it's firewall turned on.
The result: I can connect to my customer's VPN easily, without any problems. And I can't connect to my own (from the outside) at all. This is very strange as both setups are basically identical. I did run a little test, and my ISP is not blocking TCP 1701. I don't know about the UDP ports. Seems unlikely though.
As per the other thread, I tried switching my server from DHCP (with a static IP assigned by the router) to a TRUE manual IP, but that hasn't had any effect.
It's very odd how two essentially identical routers have different results using L2TP.
You've got a blocked port somewhere, or a configuration difference.
In general, don't use port-forwarding NAT for VPNs, use a gateway-firewall that contains a VPN server.
Airport and Time Capsule are client-focused devices. They're not good for use with servers and server traffic and (if you look through the forums) a common source of problems.
NAT devices remap and hide host IP end-points behind public IP one address, where VPNs expressly seek to isolate and identify those specific IP end-points; the two are working at cross purposes.
As for the debugging sequence used, see my previous posts,
Try PPTP. That works better around NAT.
Better still, get yourself the proper server-grade gateway-firewall device for the purposes of servers and of allowing remote access via VPN, and don't try to repurpose a client-oriented gateway-firewall device. Configure and connect your VPN into the gateway device, and not port-forwarded and NAT'd into a server.