I'm running 10.6.8 Server on a Mac Mini. The VPN server used to work reliably, but for months now I haven't been able to connect to it. PPTP is completely busted ("MPPE required, but keys are not available. Possible plugin problem?" error) and I've given up on it.
However, L2TP works partly. I can connect to it on my local network, but if I try to connect to it from an external connection (3G on my iPhone or café WiFi on my MacBook Pro), I can't connect. The VPN server doesn't even receive the incoming connection request. There are no errors in the log, nor even entries about attempted connetions. I'm using MSChapv2 with a shared secret and the VPN IP address range is outside that of the server and the router's DHCP addresses.
I have a Airport Extreme Base Station. The correct L2TP ports are open but for whatever reason, the VPN clients on my other devices can't establish a connection with the server.
iToaster, you got me on the right track thinking that the problem was with port forwarding. I tried to find an up-to-date Apple document about which ports their services use. Before, I had been relying on OS X 10.6 manuals and documentation and some web searches.
I found this Apple document and looked at the entries for ports 500, 1701, and 4500. Under UDP port 500, it mentions that this port is also used by Back to My Mac. But the under UDP port 4500, there was the following entry:
OS X Server VPN service, Back to My Mac.Note: Configuring Back to My Mac on an AirPort Base Station or Time Capsule in NAT mode will impede connectivity to an OS X Server VPN service behind that NAT.
I checked my Airport Extreme and noticed that my Apple ID was entered in the Back to My Mac section of the Base Station. I deleted that, rebooted the router, and tried to connect to the L2TP VPN server. This time it worked.
For some baffling reason, Apple has decided that Back to My Mac should use the same ports as L2TP VPN servers. I don't understand why they would do this. Surely there must be other ports they can use for Back to My Mac.
Well, at least the mystery is solved.