You can make a difference in the Apple Support Community!

When you sign up with your Apple Account, you can provide valuable feedback to other community members by upvoting helpful replies and User Tips.

Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Mountain Lion (10.8) certificates-based L2TP over IPsec VPN kaput

After upgrading to Mountain Lion, my certificates-based L2TP over IPsec VPN stopped working. However, it works fine using a PSK instead of certificates. OSX 10.7, as well as Windows clients have no trouble with the certificates. For OSX 10.8, the VPN server is complaining the payload is faulty. So, since this works fine for OSX 10.7 and Windows clients, I have to conclude that the Mac client is mangling the certificate payload in 10.8.


In the Mac logs, I see the same as the user in this thread: https://discussions.apple.com/thread/4139538?answerId=19058470022#19058470022 . I have also followed the suggested solution in that thread of allowing all applications access to the private key in the Keychain, to no avail, the issue persists and the logs are unchanged.


Any ideas?


Cheers.

MacBook Pro, OS X Mountain Lion

Posted on Jul 30, 2012 11:48 AM

Reply
33 replies

Aug 17, 2012 12:59 AM in response to 3g91ld3a

Hi,



Interesting detail: if I connect from the LAN side to our vpn system, the ipsec connection goes up! 😕

When I come from the WAN-Side, the connection never goes to established-state.

The error message on the server side is:

>>next payload type of ISAKMP Message has an unknown value: 132


@APPLE: you really messed something up here - please fix!

Aug 31, 2012 4:21 AM in response to 3g91ld3a

I've been able to solve this problem by copying the following three files from my last OSX 10.7 backup into the 10.8 installation:


  1. /usr/sbin/racoon
  2. /System/Library/LaunchDaemons/com.apple.racoon.plist
  3. /System/Library/Sandbox/Profiles/racoon.sb


The first file is the IPsec client binary, the second tells how to launch the client, and the third specifies what the client is allowed to do on your system.

Sep 2, 2012 12:04 PM in response to ct181

Thanks for the idea, but it doesn't work for me.


I restored those 3 files from a Lion backup and rebooted the machine. Then the problem is worse, since launchd can't even launch racoon : "racoon[431]: Configuration Parse Error. (cfparse: yyparse erred, filename /etc/racoon/racoon.conf). (failure: fatal parse failure)"


I also tried to restore racoon.conf and racoonctl from Lion, but with no success either ...

Sep 5, 2012 11:00 AM in response to christophefrom25

I ran into the same problem today while trying to fix a colleague's VPN. It turned out that he had iCloud enabled, which apparently generates a config file /var/run/racoon/<some-ipv6-address>.cfg that the old racoon doesn't understand (because it is not IPv6-ready, presumably). If you move the file out of the /var/run/racoon directory, the old client should start.


We also deactivated IPv6 on the Terminal via

networksetup -setv6off Ethernet

("Ethernet" is the network device you are using for the VPN; see networksetup -listallnetworkservices for a list). However, I am not sure if that changes anything, since we did it before moving the config file away.


And he still does have some weird problem with the server certificate not being accepted, though. On another colleague's MacBook it worked with copying the three files, and deleting and re-importing the CA certificate into the system keychain.


Hope that helps...

Sep 6, 2012 8:58 AM in response to ct181

Yes, it helps a lot ! 🙂 It (almost) works, thanks to your advices.


As you said, the problem to make the Lion version of racoon work again on Mountain Lion is the presence of those files in /var/run/racoon. (the last line of /etc/racoon/racoon.conf is 'include /var/run/racoon/*.conf')


In my case, I didn't find any .cfg file, but two .conf files. If I delete them, all is OK, but they appear again after reboot.

The first one is the one you're talking about - the "Back To My Mac" feature. I disabled it, and it's gone.


But I can't find out what is the other one. I tried the -setv6off thing on both Wi-Fi and VPN interface, but it doesn't work. I have to remove the file manually until I find the solution.


But thanks a lot for your help !

Sep 18, 2012 8:00 AM in response to 3g91ld3a

We also try, to use Cert-Based L2TP-VPN

via strongswan on 10.8.1 and also got no luck.

We have the apple-specific certificate-usage proposal (ipsec),

aswell as the DNS-name of the system built in to the certificates,

but it wont work (even without NAT/PAT).


The only thing, were not publishing is CRL's - could that be part of the solution maybe?


Anyone got CRLs published (via HTTP or elseway)?


Thanks,


Jakob

Sep 19, 2012 2:23 PM in response to 3g91ld3a

Hi, I just update my iPhone to iOS 6, and guess what - now the IPSEC VPN does not connect anymore. On the other end of the tunnel, I have a StronSwan router, showing this


> next payload type of ISAKMP Message has an unknown value: 132


error message several times before it quits with a


> max number of retransmissions (2) reached STATE_MAIN_R2.


Thanks for any hint!


Regards,


Carsten

Mountain Lion (10.8) certificates-based L2TP over IPsec VPN kaput

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.