i'm pretty sure it's conflicting with Back to My Mac. i'd been fiddling around with it for days why i couldn't establish a VPN connection on a VPN gateway (win2k8r2) behind Airport Extreme (Back to My Mac) enabled.
i thought at first it was the firewall, but it was not, then i checked the ports, i even tried PPTP, but i couldn't establish a successful connection even though i have the same VPN setup as with the one in our workplace.
i then called Apple Support about this issue, i asked what ports Back to My Mac are using, he told me a dozen of ports which includes the ports for L2TP (1701, 4500, 500). and some in PPTP ports too i think, i can't remember them all.
it would be nice though if the two (VPN and Back to My Mac) can run without conflicts. but yea... i knew already at first when i first used Back to My Mac it's basically a VPN linked to your Apple ID. simple/good implementation by Apple but i wish they used different ports right?
edit: actually it is documented, it is written somewhere in Apple Support that you have to turn off Back to My Mac/MobileMe to use VPN. i just read it the other day.
So I've found the offical word burried in an Apple Document...
scroll down to Port 4500 which is used for IPSec VPN and Back to My Mac...
4500 UDP IKE NAT Traversal - ipsec-msft Mac OS X Server VPN service, Back to My Mac (MobileMe, Mac OS X v10.5 or later).
Note: VPN and MobileMe are mutually exclusive when configured through an Apple access point (such as an AirPort Base Station); MobileMe will take precedence.
NOTE TO APPLE:
1) Do NOT create network services that conflict with well known and used TCP and UDP ports.
2) If you create two conflicting services. Please MAKE A NOTE IN THE MANUAL for OS X Server so Admins are aware of the problem.
This is still prevalent today on El Capitan. If anyone has OS X server installed, do not turn on Back To My Mac if you plan on having a VPN setup.
Doing so knocked my server out, and i had to go through a significant amount of work to fix it.
Turn off Back To My Mac, and do a kickstart
sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/k ickstart -activate -configure -access -off -restart -agent -privs -all -allowAccessFor -allUsers
Then do a
sudo launchctl stop com.apple.racoon
sudo launchctl start com.apple.racoon
and reboot, and when the server comes back up restart the VPN service through the OS X server console .