L2TP VPN Server stopped working suddenly??

Question

TiMoKi1906 Author

Level 1

20 points

L2TP VPN Server stopped working suddenly??

Ok, here the scenario:

Mac Mini Server with 10.6.5
- has fixed ip
- connected to Internet via:

Linksys WRT610N Router
- has UDP ports 500, 1701, 4500 forwared to server's fixed IP

L2TP has been set up successfully with shared secret and previously has been working fine within the local LAN as well as from external source (in this case my home).

I could connect via iPhone, iPad and iMac without any problems, but then one day without really changing anything it just stopped working on all devices with the simple message:
"VPN server is not responding please try again"

So I've done following things:
1) Internally it works with find with local IP Address (externally will pop up above message)
2) I've checked all logs and result is:
SERVER:
Nov 29 16:33:38 mac racoon[169]: Connecting.
Nov 29 16:33:38 mac racoon[169]: IKE Packet: receive success. (Responder, Main-Mode message 1).
Nov 29 16:33:38 mac racoon[169]: IKE Packet: transmit success. (Responder, Main-Mode message 2).
Nov 29 16:33:38 mac racoon[169]: IKE Packet: receive success. (Responder, Main-Mode message 3).
Nov 29 16:33:38 mac racoon[169]: IKE Packet: transmit success. (Responder, Main-Mode message 4).
Nov 29 16:33:41 mac racoon[169]: IKE Packet: transmit success. (Phase1 Retransmit).
Nov 29 16:33:47 mac racoon[169]: IKE Packet: transmit success. (Phase1 Retransmit).
Nov 29 16:34:08 mac racoon[169]: IKEv1 Phase1: maximum retransmits. (Phase1 Maximum Retransmits).
Nov 29 16:34:08 mac racoon[169]: Disconnecting. (Connection tried to negotiate for, 30.293015 seconds).
Nov 29 16:34:08 mac racoon[169]: IKE Phase1 Failure-Rate Statistic. (Failure-Rate = 100.000).

CLIENT:
29/11/2010 4:33:38 PM racoon[263] Connecting.
29/11/2010 4:33:38 PM racoon[263] IKE Packet: transmit success. (Initiator, Main-Mode message 1).
29/11/2010 4:33:38 PM racoon[263] IKE Packet: receive success. (Initiator, Main-Mode message 2).
29/11/2010 4:33:38 PM racoon[263] IKE Packet: transmit success. (Initiator, Main-Mode message 3).
29/11/2010 4:33:38 PM racoon[263] IKE Packet: receive success. (Initiator, Main-Mode message 4).
29/11/2010 4:33:38 PM racoon[263] IKE Packet: transmit success. (Initiator, Main-Mode message 5).
29/11/2010 4:33:38 PM racoon[263] IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
29/11/2010 4:33:41 PM racoon[263] IKE Packet: transmit success. (Phase1 Retransmit).
29/11/2010 4:33:41 PM racoon[263] IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
29/11/2010 4:33:41 PM racoon[263] IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
29/11/2010 4:33:44 PM racoon[263] IKE Packet: transmit success. (Phase1 Retransmit).
29/11/2010 4:33:44 PM racoon[263] IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
29/11/2010 4:33:44 PM racoon[263] IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
29/11/2010 4:33:47 PM racoon[263] IKE Packet: transmit success. (Phase1 Retransmit).
29/11/2010 4:33:47 PM racoon[263] IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
29/11/2010 4:33:47 PM racoon[263] IKE Packet: receive failed. (Initiator, Main-Mode Message 6).
and so on

I've went through all steps again and again:
- en/disabling VPN passthrough on the router
- en/disabling UDP port forwarding on the router
- reboot router
- remove VPN Service and add VPN Service again
- start/stop VPN Service
- switch of Firewall completely on Mac Server

I'm really stucked here. Especially it was working before and with no reason - it just stopped. First thought might be iOS 4.2 update, but then it should still work in my iMac and Macbook but it doesn't.

I've googled a lot and searched all threads about this here and tried suggestions but none of them are working.

Hope anybody can help out.

Thanks a lot in advance.

Mac Mini Server, Mac OS X (10.6.5), L2TP VPN IPsec

Posted on Nov 29, 2010 1:50 AM

Reply

Answer 1

Best reply

kuantize

Level 1

10 points

Nov 29, 2010 6:28 AM in response to TiMoKi1906

This happened to me once. For the sake of argument lets say you have all the correct ports forwarded on your router to your OSX Server.

- I have never been able to connect with the same VPN user account twice, so check to see if you have a stale session logged in on the VPN control panel on OSX Server
- When I used to get that message it was an issue with the shared secret, if you are using L2TP with IPSEC. Try changing to shared secret temporarily to something easier (ie, 8 characters, letters and numbers) and see if you can log in that way.

With the password issue, I do not know if the keychain service has something to do with it, but I am sure that it messed up the VPN shared secret the first time.

tl;dr

Try changing to VPN shared secret passphrase to something easier, test to see if you can log in, and if that is the case your problem is: the shared secret password.

If not, check to see if you have any stale sessions (accounts logged in to the VPN that for some reason have not timed out)

Regards,

Reply

Answer 2

TiMoKi1906 Author

Level 1

20 points

Nov 29, 2010 7:16 PM in response to kuantize

Well I've tried that already. Changed the shared secret. As I mentioned I even removed the entire VPN service, so all sessions should be terminated.

But here comes now the weird part:

Just for the the fun of it, I switched the VPN pass through off and on again on the router (I've done exactly the same thing several times before without any effect), but now it works again.

I've now idea why. Because believe me, I've done exactly the same thing half day before and it didn't have any effect.

I keep in observing but at least for now it seems that somehow the router caused the problem, even though I still don't know how, why and how come its working now again.

I'll keep it updated, as I still don't find that this is really a 'solution' rather than a random act.

Reply

Answer 3

TiMoKi1906 Author

Level 1

20 points

Nov 30, 2010 4:22 AM in response to TiMoKi1906

And now it stopped working again for no reason. I've done literally nothing on any settings. Only time has passed by and I've exactly the same problem again as described above.

I've tried my previous "solution" with switching off/on tahr VPN pass through again on the router, but no result so far.

I still think it's the somehow the problem but I cant explain why first it just stop working after 24hrs. And only for VPN, all other services such as email, FTP, http, etc which are also being forwarded by the same router to the same server are all working fine.

Hope anybody has some bright ideas because I have none at this moment.

Message was edited by: TiMoKi1906

Reply

Answer 4

xjrguy

Level 1

0 points

Dec 1, 2010 6:28 PM in response to TiMoKi1906

Make sure that in System Preferences, in your Network Control panel that your interfaces are prioritized properly.

For example, if you have Bluetooth or Airport on top of Ethernet and that is how you have the mini VPN server connected to the Internet (via Ethernet) I've found it will break unless it is at the top,

Hope that helps

Reply

Answer 5

TiMoKi1906 Author

Level 1

20 points

Dec 2, 2010 1:26 AM in response to xjrguy

Thanks for the tip. I've checked.. but on the server 'Ethernet' is already on the first position.

I still also think its the Wifi-Router issue. Because if I connect to the VPN via our internal network (using local IP, not public fixed IP) then it works fine. Meaning the VPN Server itself seems to be responding fine and is configured correctly.

Only when I use public IP (which will be re-directed through the Wifi-Router via LAN to the server) then I have problems.

It just seems odd that I can make it work for a day somehow (with random changes back and forth on the router) but then it will just stop after 24hours without doing anything.

Still strange to me.... wondering if changing the router firmware to DD-WRT would help.

Reply

Answer 6

TiMoKi1906 Author

Level 1

20 points

Dec 6, 2010 2:51 AM in response to TiMoKi1906

So I've flashed my router with dd-wrt and problem is solved. I can connect to my VPN fine from all devices at all locations. Internally and externally. Seems in the original linksys firmware is some bug.

Have observed the situation now for several days and eveythings works fine. Luckily there exist dd-wrt, which also added some nice extra features to my router.

Thanks everybody here for their previous help. And in case somebody else has same problem and need help, feel free to contact me.

Reply

Answer 7

Mitnick

Level 1

4 points

Jan 6, 2011 2:32 PM in response to TiMoKi1906

Did have the same problems, all ports (50,500,1701,1723,4500) forwarded to the correct ip-address and nothing worked from the outside.

A firmware upgrade of my Cisco Linksys E3000 to v1.003 resolved all my problems!

Reply

Answer 8

Jan 20, 2011 10:18 AM in response to TiMoKi1906

I have a similar problem with my VPN server on an Xserve running 10.6.5 with a DIR-655 router. Users try to connect and get no response from the server. Stopping and restarting the VPN service fixes the problem and allows users to login again. This seems to happen randomly. Looking for a solution to solve this problem or for ideas on where to look for issues.

Reply