Previous 1 2 3 Next 33 Replies Latest reply: Feb 13, 2014 4:08 PM by tqxw
3g91ld3a Level 1 Level 1

After upgrading to Mountain Lion, my certificates-based L2TP over IPsec VPN stopped working. However, it works fine using a PSK instead of certificates. OSX 10.7, as well as Windows clients have no trouble with the certificates. For OSX 10.8, the VPN server is complaining the payload is faulty. So, since this works fine for OSX 10.7 and Windows clients, I have to conclude that the Mac client is mangling the certificate payload in 10.8.


In the Mac logs, I see the same as the user in this thread: . I have also followed the suggested solution in that thread of allowing all applications access to the private key in the Keychain, to no avail, the issue persists and the logs are unchanged.


Any ideas?



MacBook Pro, OS X Mountain Lion
Reply by haraldfromenns on Sep 22, 2012 12:33 AM Helpful

Hi to all...

Problem solved! Because OSX and IOS have the same problem, i dived into the deep of IKE and found

the solution.

The problem is, that the new IPSEC system has problems with handling "big" certificates (search for

IKE UDP fragmentation, if you are interested in).

So, the solution is quiet simple- create a certificate with the absolute MINIMUM of the required data

(for Example: C=AT, CN=HS, E=hs). I tested only with 1024Bit Public Key size- and this works on

IOS and OSX as well.


Habe fun,



All replies

  • haraldfromenns Level 1 Level 1



    Have the same issue. IPSec stopped working after the f+++ mountain loion update.

    I did a lot of debug and research - no solution yet.


    Any news from your side?

  • 3g91ld3a Level 1 Level 1

    No news. I spoke with Enterprise support and they indicated there was no one, anywhere, that could help me with the VPN client, and, therefore, escalate the issue to Engineering.


    So, my only hope at this point is that PM or Eng is monitoring the forums, or, someone finds a viable workaround that still uses certificates.

  • beamzz Level 1 Level 1



    I also have the exact same problem!!

    I am surprised that there is no official comment yet from Apple..

  • haraldfromenns Level 1 Level 1




    Interesting detail: if I connect from the LAN side to our vpn system, the ipsec connection goes up!

    When I come from the WAN-Side, the connection never goes to established-state.

    The error message on the server side is:

    >>next payload type of ISAKMP Message has an unknown value: 132


    @APPLE: you really messed something up here - please fix!

  • christophefrom25 Level 1 Level 1



    I have exactly the same problem. Could you tell me more about using PSK instead of certificates ? How do you do this ? Is there something to change on server side ?


    Thanks a lot for your answers.

  • 3g91ld3a Level 1 Level 1

    Note for any watching this thread: I upgraded to 10.8.1 and the issue persists. The VPN server still complains of a mangled certificate payload.

  • ct181 Level 1 Level 1

    I've been able to solve this problem by copying the following three files from my last OSX 10.7 backup into the 10.8 installation:


    1. /usr/sbin/racoon
    2. /System/Library/LaunchDaemons/
    3. /System/Library/Sandbox/Profiles/


    The first file is the IPsec client binary, the second tells how to launch the client, and the third specifies what the client is allowed to do on your system.

  • christophefrom25 Level 1 Level 1

    Thanks for the idea, but it doesn't work for me.


    I restored those 3 files from a Lion backup and rebooted the machine. Then the problem is worse, since launchd can't even launch racoon : "racoon[431]: Configuration Parse Error. (cfparse: yyparse erred, filename /etc/racoon/racoon.conf). (failure: fatal parse failure)"


    I also tried to restore racoon.conf and racoonctl from Lion, but with no success either ...

  • ct181 Level 1 Level 1

    I ran into the same problem today while trying to fix a colleague's VPN.  It turned out that he had iCloud enabled, which apparently generates a config file /var/run/racoon/<some-ipv6-address>.cfg that the old racoon doesn't understand (because it is not IPv6-ready, presumably).  If you move the file out of the /var/run/racoon directory, the old client should start.


    We also deactivated IPv6 on the Terminal via

         networksetup -setv6off Ethernet

    ("Ethernet" is the network device you are using for the VPN; see networksetup -listallnetworkservices for a list).  However, I am not sure if that changes anything, since we did it before moving the config file away.


    And he still does have some weird problem with the server certificate not being accepted, though.  On another colleague's MacBook it worked with copying the three files, and deleting and re-importing the CA certificate into the system keychain.


    Hope that helps...

  • ct181 Level 1 Level 1

    PS: The foo:bar:foo:foo.conf file is created by the "Back to My Mac" feature – if you don't use this, you could disable it in the iCould System Preference Pane and the conf file will be automatically removed.

  • christophefrom25 Level 1 Level 1

    Yes, it helps a lot ! It (almost) works, thanks to your advices.


    As you said, the problem to make the Lion version of racoon work again on Mountain Lion is the presence of those files in /var/run/racoon. (the last line of /etc/racoon/racoon.conf is 'include /var/run/racoon/*.conf')


    In my case, I didn't find any .cfg file, but two .conf files. If I delete them, all is OK, but they appear again after reboot.

    The first one is the one you're talking about - the "Back To My Mac" feature. I disabled it, and it's gone.


    But I can't find out what is the other one. I tried the -setv6off thing on both Wi-Fi and VPN interface, but it doesn't work. I have to remove the file manually until I find the solution.


    But thanks a lot for your help !

  • Jakob Tewes Level 1 Level 1

    We also try, to use Cert-Based L2TP-VPN

    via strongswan on 10.8.1 and also got no luck.

    We have the apple-specific certificate-usage proposal (ipsec),

    aswell as the DNS-name of the system built in to the certificates,

    but it wont work (even without NAT/PAT).


    The only thing, were not publishing is CRL's - could that be part of the solution maybe?


    Anyone got CRLs published (via HTTP or elseway)?





  • cpohle Level 1 Level 1

    Hi, I just update my iPhone to iOS 6, and guess what - now the IPSEC VPN does not connect anymore. On the other end of the tunnel, I have a StronSwan router, showing this


    > next payload type of ISAKMP Message has an unknown value: 132


    error message several times before it quits with a


    > max number of retransmissions (2) reached STATE_MAIN_R2.


    Thanks for any hint!





  • Yukiru Level 1 Level 1


    Yes same here, tried it today too. IOS5 works, IOS6 doesnt. Apple keeps on screwing our infrastructure.

    If any engineer from apple looks at this post, please change it back, so it will work again.


    And exact the same problem like in OS X 10.8. VPN stopps connecting after phase 1.

Previous 1 2 3 Next