VPN not connecting with no obvious error

Question

Level 1

0 points

VPN not connecting with no obvious error

I just setup VPN on our Lion Server (10.7.4) using the instructions here. I've tried from both my Mac and my iPhone, and neither succesfully connect. The only message it displays is "VPN Server not responding", even though the log says transmit success. Any thoughts?

May 21 10:36:27 Atomic1 racoon: IKE Packet: receive success. (Responder, Main-Mode message 1).

May 21 10:36:27 Atomic1 racoon: IKE Packet: transmit success. (Responder, Main-Mode message 2).
May 21 10:36:27 Atomic1 racoon: [9726] INFO: Hashing 555.444.3.222[500] with algo #2
May 21 10:36:27 Atomic1 racoon: [9726] INFO: NAT-D payload #0 doesn't match
May 21 10:36:27 Atomic1 racoon: [9726] INFO: Hashing 555.444.3.222[1357] with algo #2
May 21 10:36:27 Atomic1 racoon: [9726] INFO: NAT-D payload #1 doesn't match
May 21 10:36:27 Atomic1 racoon: [9726] INFO: NAT detected: ME PEER
May 21 10:36:27 Atomic1 racoon: IKE Packet: receive success. (Responder, Main-Mode message 3).
May 21 10:36:27 Atomic1 racoon: [9726] INFO: Hashing 555.444.3.222[1357] with algo #2
May 21 10:36:27 Atomic1 racoon: [9726] INFO: Hashing 555.444.3.222[500] with algo #2
May 21 10:36:27 Atomic1 racoon: [9726] INFO: Adding remote and local NAT-D payloads.
May 21 10:36:27 Atomic1 racoon: IKE Packet: transmit success. (Responder, Main-Mode message 4).
May 21 10:36:30 Atomic1 racoon: IKE Packet: transmit success. (Phase1 Retransmit).
May 21 10:36:30 Atomic1 racoon: IKE Packet: transmit success. (Phase1 Retransmit).
May 21 10:36:33 Atomic1 racoon: IKE Packet: transmit success. (Phase1 Retransmit).
May 21 10:36:36 Atomic1 racoon: IKE Packet: transmit success. (Phase1 Retransmit).
May 21 10:36:48 Atomic1 racoon: IKE Packet: transmit success. (Phase1 Retransmit).
May 21 10:36:48 Atomic1 racoon: IKE Packet: transmit success. (Phase1 Retransmit).

Posted on May 21, 2012 7:42 AM

Reply

Answer 1

May 21, 2012 8:04 PM in response to woz.was

I had same situation. I didn't touch anything about VPN setup recently and had no clue to fix it. After googling, I found that "repair permission" with DiskUtility can fix it in Japanese blog (without obvious reason). I tried and succeeded. Please try!

Reply

Answer 2

May 22, 2012 8:54 AM in response to woz.was

INFO: NAT-D payload #1 doesn't match

Are you forwarding IKE NAT Traversal from your router to your server? UDP 500

Reply

Answer 3

woz.was Author

Level 1

0 points

May 22, 2012 10:36 AM in response to yoshihito

I'm trying the repair permissions now, but I did confirm that the NAT 500 port has been open with UDP settting. Should I need to reset/powercycle my modem and router after making the initial port settings?

Reply

Answer 4

woz.was Author

Level 1

0 points

May 22, 2012 12:33 PM in response to woz.was

Nope, repair permissions was no good. Still not working with same messages.

Reply

Answer 5

May 22, 2012 12:43 PM in response to woz.was

What are all of the services that you are port-forwarding from your router?

Reply

Answer 6

woz.was Author

Level 1

0 points

May 22, 2012 12:46 PM in response to Jonathan Melville

Reply

Answer 7

May 22, 2012 12:49 PM in response to woz.was

You are forwarding 1701 (L2TP) over TCP but L2TP is actually UDP.

Reply

Answer 8

woz.was Author

Level 1

0 points

May 22, 2012 1:08 PM in response to Jonathan Melville

Changed it, same messages. I haven't reset the router since making those changes, should that matter?

Reply

Answer 9

May 22, 2012 1:09 PM in response to woz.was

You really shouldn't have to, but I guess it's worth a try.

Is everything else forwarding correctly? Can you SSH into your server from the internet?

Reply

Answer 10

woz.was Author

Level 1

0 points

May 22, 2012 1:11 PM in response to Jonathan Melville

Yea, everything else still works great.

Reply

Answer 11

Jul 6, 2012 10:34 AM in response to woz.was

Back to My Mac shares one of the ports with the VPN service. It's not documented very well but it takes priority over VPN. Try disabling Back to My Mac on the server, any Macs on the network, and any airport express or airport extreme on the network. I have the exact same problem, and this fixed it.

Reply

Answer 12

bfdulock

Level 2

214 points

Jul 6, 2012 6:49 PM in response to woz.was

Make sure you allow IPSec passthrough on your router (if it gives you this option).

Reply

Answer 13

woz.was Author

Level 1

0 points

Jul 9, 2012 9:01 AM in response to bfdulock

Thanks for the posts guys, but no luck in either case. IPSec is enabled, and no one is using Back to My mac. Wish it had been something that simple.

Reply

Answer 14

bfdulock

Level 2

214 points

Jul 12, 2012 4:52 PM in response to woz.was

Try connecting to the VPN on your local network (i.e., bypass the router). If it works, the problem is very likely with the router configuration. If it does not work, then try the Terminal command described in this article:

http://support.apple.com/kb/HT4748?viewlocale=en_US&locale=en_US

NOTE: the KB article applies to both PPTP and L2TP over IPSec even though it seems to only apply to PPTP.

Reply

Answer 15

woz.was Author

Level 1

0 points

Jul 17, 2012 6:56 AM in response to bfdulock

I've never done that, so it could definitely help. The problem is, when I go to follow the instructions, I can't find VPN key agent user, or anything similar, in either Server App or Workgroup manager. Could the user not exist? If so, what's the best way to create it?

Reply