ISA Server L2TP VPN with Certifcates

Hi all

Hopefully someone older/wiser can help me with this issue. I am trying to connect my OS 10.4.3 machine to a windows ISA Server using VPN. The catch is I am trying to get it to work using certifactes for machine authentication rather than a pre-shared key (just for that added bit of paranoia)

The VPN server is set up and works like a charm with my windows xp laptop certificates et all! When i set up Internet Connect on the mac to use a pre-shared key for authentication it also works no problems.

So i exported my certificate along with private key from the xp client machine to a pkcs#12 file and imported it into the system keychain on the mac (dragging the CA certificate to the X509Anchors keychain) and all is well - the certificate registers as valid and i am able to select it for machine authentication in my internet connect preferences.

However, when i click connect on the mac the connection times out just as it would if i had entered an incorrect pre-shared key as the machine authentication method. Can anyone tell me if this is an issue of compatibility between certificates issued by a windows based certification authority, or if I am doing something wrong somewhere. Has anyone successfully linked a tiger vpn client to an ISA server using certificate based machine authentication or am i just going to have to settle for PSK authentication.

Thanks

Apple Developer Platform 2,1, Mac OS X (10.4.3), Build 8F1099

Posted on Jan 9, 2006 2:32 AM

Reply
4 replies

Mar 15, 2006 7:17 PM in response to Ouch!Potato

I have the exact same issues. I'm using Internet Connect on a OS X 10.4.5 client and the VPN server is running OS X Server 10.4.5. I generated and signed the certificate on the VPN server. I get the exact same result: connection simply times out and it says that it couldn't connect to the server.

I believe I've opened up all relevant ports on the non-OS X firewall in between and my ISP doesn't block any ports. Connections work perfectly using a pre-shared key.

I'm about to give up on certificates.

May 31, 2006 2:31 PM in response to Ouch!Potato

I'm having the same problem. I've got a machine cert on my Mac OS 10.4.6 client that was issued by my Win2003 CA. When I try and connect, it just hangs and then dies. In the Security Logs on the 2003 L2TP server, I even see a successful IKE negotiation (MS Event ID 541 and 543 below).

EventID 541:
IKE security association established.
Mode:
Key Exchange Mode (Main Mode)

Peer Identity:
Certificate based Identity.
Peer Subject C=US, S=City, L=State, O=Company, OU=group, CN=machine.subdomain.company.com, E=me@company.com
Peer SHA Thumbprint peerthumbrint
Peer Issuing Certificate Authority O=company.com, CN=Certificate Authority
Root Certificate Authority O=company.com, CN=Certificate Authority
My Subject CN=server.subdomain.company.com
My SHA Thumbprint mythumbrint
Peer IP Address: x.x.x.x

Filter:
Source IP Address x.x.x.x
Source IP Address Mask 255.255.255.255
Destination IP Address x.x.x.x
Destination IP Address Mask 255.255.255.255
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr x.x.x.x
IKE Peer Addr x.x.x.x
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr

Parameters:
ESP Algorithm Triple DES CBC
HMAC Algorithm SHA
Lifetime (sec) 3600
MM delta time (sec) 1


EventID 543:
IKE security association ended.
Mode: Key Exchange (Main mode)
Filter:
Source IP Address X.X.X.X
Source IP Address Mask 255.255.255.255
Destination IP Address X.X.X.X
Destination IP Address Mask 255.255.255.255
Protocol 0
Source Port 0
Destination Port 0
IKE Local Addr X.X.X.X
IKE Peer Addr X.X.X.X
IKE Source Port 500
IKE Destination Port 500
Peer Private Addr

At least give me a some methods to debug with.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

ISA Server L2TP VPN with Certifcates

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.