PPTP isn't an option, unfortunately. It's not considered secure enough by the folks who run the VPN server end of things.
I managed to figure out what part of the GUI plumbing is responsible for the /var/run/racoon/ configuration. It comes from the /System/Library/Extensions/L2TP.ppp/Contents/MacOS/L2TP executable. It does NOT come from /usr/sbin/vpnd (though they share enough code for things to be confusing).
There's no obvious configuration file that lets me get at the options it sets. But, in a stroke of luck, one of the racoon.conf options baked into the executable is a) long and b) unnecessary. The "situation identity_only" directive is pointless if I believe the man page. So, I mv'ed the original L2TP executable asked and used a hex editor to replace "situation identity_only" with "nat_traversal force" (using approrpriate excess padding so everything stays the same size).
Good news: I can get the "nat_traversal force" directive in the dynamic racoon.conf. I didn't break anything by hacking the L2TP binary.
Bad news: Forcing NAT-T with that directive doesn't appear to work. Even with that directive in place, racoon still tries to do ESP/IP even when it shouldn't/can't. <sigh>
Another thing I explored was replacing the old racoon with a more modern racoon, but modern racoon doesn't support the keychain integration AFAICT.
Hey, I am facing a Nat-t problem with os x integrated ipsec client myself, and your post looks interesting to me!
I have read that nat-t(raversal) parameter is a compile time parameter, so modifying the executable was quite a good idea
I was wondering how did you proceed to modify L2TP.ppp ?
I can edit it using an hex editor, but I can't find a way to locate the nat-t parameter...
Can you help ?
Modifying this binary seems to be the only solution, and if any problem occurs, having a backup copy to restore would not imply any os reinstallation...
If I find another solution I would be happy, but it's been almost a week that I am trying to configure a OS X 10.8.2 ipsec connection to a cisco RV180 router... without success.
The only answer I had from cisco was to use a third party client.... it is a shame.
Finding out that os x embeded cisco ipsec client doesn't work with a small business cisco router is quite a shame also (especially without any possibility to tweak client side parameters).
Why calling this vpn client Cisco ipsec ???
I am really disapointed on both sides for now.
Although this question may seem related to yours, it is really different. I suggest you start your own question. The original poster's VPN was just misconfigured. Considering that most VPN systems are run by corporate IT, about 94% of them are probably misconfigured. Not much you can do about that.
PS: Do NOT modify your binaries