Mac VPN to L2TP on IPSec Connection Issues

My company has implemented a VPN using Microsoft’s VPNs for Windows Server 2003. Recently they’ve changed from PPTP to L2TP over IPSec. I have yet to be able to configure the Macintosh to use the new protocol VPN. It worked perfectly with PPTP. Included below is a link to the documentation on the VPN server that my company is using.

http://www.microsoft.com/vpn

I’m trying to access this VPN on a Macintosh Running Tiger 10.4.2 using it’s built in VPN Client Software (Internet Connect). Here are the steps I’ve taken thus far.

Opened the application “Internet Connect”.
Selected the New VPN Connection from the file menu.
Selected the radio button for “L2TP over IPSec”.
From the resulting window, selected “Edit Configuration” from the “Configuration” drop down list.
From the resulting configuration window, enter the following values:
Description: Dart VPN
Server Address: VPN's domain name
Account Name: my user id

Now, my company has supplied me with a certificate (VPNUserCert.pfx) and a certificate authority (dartca.cer). So using the application “Keychain Access” I did the following:

Selected the “Import...” menu item from the “File” menu.
From the resulting dialog box:
From the file chooser, selected the “dartca.cer” file.
From the “Keychain:” drop down list, selected “X509Anchors”.
Clicked on “Open”.
Entered my username (administrator) and password in the resulting “Authenticate” window.
Restarted application “KeyChain Access”.
Selected the “Import...” menu item from the “File” menu.
From the resulting dialog box:
From the file chooser, selected the “VPNUserCert.pfx” file.
From the “Keychain:” drop down list, selected “login”.
Clicked on “Open”.
Clicked on “Okay” from the resulting “Authenicate” window without entering a password, since the file is not password protected.

Now with the “login” keychain selected the “MY COMANY NAME Employee” certificate was visible. Selecting that certificate showed the following:

With the “X509Anchors” keychain selected the “MY COMPANY NAME CA” certificate authority was visible. Selecting it showed the following:


Returning to the application “Internet Connect” and continuing from “Edit configuration” window, I did the following:
User Authentication: checked the “Certificate” radio button.
From the resulting “Select Certificate” dialog box I chose (my only choice) “MY COMPANY NAME Employee”.
Clicked “Okay” to accept.
Since, my company doesn’t provide a machine certificate or a shared secret I left that field blank

Choosing connect resulted in the following message in the console log and an error from “Internet Connect” saying, “The connection has failed. Please verify your settings and try again.”

===== Wednesday, September 14, 2005 1:41:37 PM America/Detroit =====
Sep 14 13:47:22 judah pppd[651]: L2TP: no user shared secret found.

It seems that “Internet Connect” either requires a “Shared Secret” or a “Machine Certificate”, but my company doesn’t use either. Of course if I select certificate for “Machine Authentication” I get the following error message:

No machine certificates found
Certificate authentication cannot be used because your keychain does not contain any suitable certificates. Use Keychain Access to import the appropriate certificates into your keychain. If you do not have the certificates required for authentication, contact your network administrator.

From Windows XP running on Virtual PC 7.0.2 on this same Macintosh following the directions at the above URL I was able to setup a VPN Connection and connect to our company’s VPN just fine. The difference being that it did not need a shared secret or a machine certificate, and that it required my User certificate and my domain password whereas in Internet Connect it required either a password or a user certificate but not both. Windows required both.

Has anyone been able to get the built in VPN client in Tiger to work using Layer

Posted on Sep 14, 2005 12:20 PM

Reply
6 replies

Sep 16, 2005 11:13 PM in response to Michael Tennes

First, Windows and OS X L2TP VPN isn't fully compatible. OS X VPN handles NAT-T differently than what Windows does so if either the client or server is behind NAT there is a possibility you never will be able to connect using Apples built-in VPN client.

Then, the VPN client should be able to find the certificate from the VPN client setup if it's imported correctly, and it looks like it's used instead of the shared secret. Never used that myself though.

I don't know if using VPNTracker instead would help in this situation.

Sep 19, 2005 7:59 AM in response to Leif Carlsson

From my computer that's not using NAT it does not work. Internet Connect only allows me to select a user certificate, but I don't know what's it's expecting for the machine authentication portion of the configuration setup.

For my machine at home sitting behind and Airport Extreme Base Station, it won't work either from Mac OS X, but it will work running on Windows XP that's runing on VIRTUAL PC 7.0.2 on the same Mac! So whatever NATing that going on there with the base station doesn't affect things.

Sep 19, 2005 8:42 AM in response to Michael Tennes

Here's the servers failure message:
Event Type: Failure Audit
Event Source: Security
Event Category: Logon/Logoff
Event ID: 546
Date: 9/7/2005
Time: 3:20:53 PM
User: NT AUTHORITY\NETWORK SERVICE
Computer: VPN-2K31
Description:
IKE security association establishment failed because peer sent invalid proposal.
Mode:
Key Exchange Mode (Main Mode)

Filter:
Source IP Address 10.192.0.27
Source IP Address Mask 255.255.255.255
Destination IP Address 4.229.198.137
Destination IP Address Mask 255.255.255.255
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr 10.192.0.27
IKE Peer Addr 4.229.198.137
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr

Attribute:
Authentication Method
Expected value:
RSA Signature with Certificates
Received value:
Preshared Key

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Sep 19, 2005 3:15 PM in response to Michael Tennes

The last part:

"Authentication Method
Expected value:
RSA Signature with Certificates
Received value:
Preshared Key"

seems to indicate it's the machine autentication that should use the certificate not the user authentication (in OS X L2TP client setup). And possibly SecureID for the user auth (or just name/password).

But looking at the error message it seems the server is behind NAT and that will surely prevent OS X built-in L2TP client to work (NAT-T negotiation different).

Sep 22, 2005 2:03 PM in response to Michael Tennes

OS X isn't compatible with Windows L2TP under NAT-T since Apple made its own implementation of the NAT-T negotiation (Mac only).

It used to work (early Panther revisions) if only the Windows machine (client) was behind NAT if ESP got through all the way. Now NAT-T is always negotiated if either server or client is behind NAT.

If both have public IP L2TP works Windows - Mac (no NAT-T negotiation).

If anything has changed lately it might work, but I doubt it.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Mac VPN to L2TP on IPSec Connection Issues

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.