Ok, my problems with VPN are not likely DNS related. I did not have my computer using my mac mini as its DNS server, so it was getting routed outside my home network.
I should note, my network configuration. Mavericks Laptop trying to connect to Mavericks mac mini server, router is a Time Capsule.
I have tried the suggestions that have worked for others in this thread without any luck, such as creatign a new user or editing/deleting systctl.conf etc. NoNe of these ideas have worked in my situation. 😟
Everything had been working yesterday. The only thing that changed was that I upgraded both the laptop and the server to Mavericks.
Comparing logs between when I can connect locally vs. using my public IP things look very different. Here is server and client side logging when I try to connect through my public IP, but both computers are on the same local network:
Server
10/26/2013 10:14:59.020 PM racoon[278]: Connecting.
10/26/2013 10:14:59.020 PM racoon[278]: IPSec Phase 1 started (Initiated by peer).
10/26/2013 10:14:59.020 PM racoon[278]: IKE Packet: receive success. (Responder, Main-Mode message 1).
10/26/2013 10:14:59.021 PM racoon[278]: >>>>> phase change status = Phase 1 started by us
10/26/2013 10:14:59.022 PM racoon[278]: IKE Packet: transmit success. (Responder, Main-Mode message 2).
10/26/2013 10:14:59.029 PM racoon[278]: IKE Packet: receive success. (Responder, Main-Mode message 3).
10/26/2013 10:14:59.050 PM racoon[278]: IKE Packet: transmit success. (Responder, Main-Mode message 4).
10/26/2013 10:14:59.050 PM racoon[278]: Connecting.
10/26/2013 10:15:02.320 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:05.619 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:08.919 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:20.919 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:37.412 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:57.205 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:16:20.303 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:16:46.641 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:17:13.641 PM racoon[278]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:17:46.637 PM racoon[278]: IKEv1 Phase 1: maximum retransmits. (Phase 1 Maximum Retransmits).
10/26/2013 10:17:46.638 PM racoon[278]: Phase 1 negotiation failed due to time up. 667e2993842c2b71:19d777ce935a8344
Client
10/26/2013 10:14:58.848 PM pppd[1560]: publish_entry SCDSet() failed: Success!
10/26/2013 10:14:58.848 PM pppd[1560]: publish_entry SCDSet() failed: Success!
10/26/2013 10:14:58.848 PM pppd[1560]: pppd 2.4.2 (Apple version 727.1.15) started by matti, uid 501
10/26/2013 10:14:59.108 PM pppd[1560]: L2TP connecting to server '----.----.net' (108.162.---.---)...
10/26/2013 10:14:59.109 PM pppd[1560]: IPSec connection started
10/26/2013 10:14:59.124 PM racoon[1561]: accepted connection on vpn control socket.
10/26/2013 10:14:59.124 PM racoon[1561]: Connecting.
10/26/2013 10:14:59.125 PM racoon[1561]: IPSec Phase 1 started (Initiated by me).
10/26/2013 10:14:59.125 PM racoon[1561]: IKE Packet: transmit success. (Initiator, Main-Mode message 1).
10/26/2013 10:14:59.125 PM racoon[1561]: >>>>> phase change status = Phase 1 started by us
10/26/2013 10:14:59.129 PM racoon[1561]: >>>>> phase change status = Phase 1 started by peer
10/26/2013 10:14:59.129 PM racoon[1561]: IKE Packet: receive success. (Initiator, Main-Mode message 2).
10/26/2013 10:14:59.134 PM racoon[1561]: IKE Packet: transmit success. (Initiator, Main-Mode message 3).
10/26/2013 10:14:59.141 PM racoon[1561]: IKE Packet: receive success. (Initiator, Main-Mode message 4).
10/26/2013 10:14:59.155 PM racoon[1561]: IKE Packet: transmit success. (Initiator, Main-Mode message 5).
10/26/2013 10:15:02.294 PM racoon[1561]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:02.427 PM racoon[1561]: !!! skipped retransmitting frags: frag_flags 1, r->sendbuf->l 112, max 1280
10/26/2013 10:15:02.427 PM racoon[1561]: Received retransmitted packet from 108.162.---.---[500].
10/26/2013 10:15:02.427 PM racoon[1561]: the packet is retransmitted by 108.162.---.---[500].
10/26/2013 10:15:05.519 PM racoon[1561]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:05.727 PM racoon[1561]: !!! skipped retransmitting frags: frag_flags 1, r->sendbuf->l 112, max 1280
10/26/2013 10:15:05.727 PM racoon[1561]: Received retransmitted packet from 108.162.---.---[500].
10/26/2013 10:15:05.727 PM racoon[1561]: the packet is retransmitted by 108.162.---.---[500].
10/26/2013 10:15:08.799 PM racoon[1561]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:09.026 PM racoon[1561]: !!! skipped retransmitting frags: frag_flags 1, r->sendbuf->l 112, max 1280
10/26/2013 10:15:09.026 PM racoon[1561]: Received retransmitted packet from 108.162.---.---[500].
10/26/2013 10:15:09.026 PM racoon[1561]: the packet is retransmitted by 108.162.---.---[500].
10/26/2013 10:15:21.026 PM racoon[1561]: !!! skipped retransmitting frags: frag_flags 1, r->sendbuf->l 112, max 1280
10/26/2013 10:15:21.026 PM racoon[1561]: Received retransmitted packet from 108.162.---.---[500].
10/26/2013 10:15:21.027 PM racoon[1561]: the packet is retransmitted by 108.162.---.---[500].
10/26/2013 10:15:21.601 PM racoon[1561]: IKE Packet: transmit success. (Phase 1 Retransmit).
10/26/2013 10:15:29.131 PM pppd[1560]: IPSec connection failed
10/26/2013 10:15:29.132 PM racoon[1561]: IPSec disconnecting from server 108.162.---.---
10/26/2013 10:15:29.133 PM racoon[1561]: glob found no matches for path "/var/run/racoon/*.conf"
10/26/2013 10:20:08.849 PM configd[18]: SCNC: start, triggered by (185) SystemUIServer, type L2TP, status 0, trafficClass 0
Seems like the server is looking for something, but the client decides not to send. Client side looks very different whe using local IP
Client (local)
10/26/2013 10:20:08.866 PM pppd[1579]: publish_entry SCDSet() failed: Success!
10/26/2013 10:20:08.866 PM pppd[1579]: publish_entry SCDSet() failed: Success!
10/26/2013 10:20:08.866 PM pppd[1579]: pppd 2.4.2 (Apple version 727.1.15) started by matti, uid 501
10/26/2013 10:20:08.868 PM pppd[1579]: L2TP connecting to server '192.168.10.120' (192.168.10.120)...
10/26/2013 10:20:08.868 PM pppd[1579]: IPSec connection started
10/26/2013 10:20:08.875 PM racoon[1580]: lifetime = 3600
10/26/2013 10:20:08.875 PM racoon[1580]: lifebyte = 0
10/26/2013 10:20:08.875 PM racoon[1580]: encklen=256
10/26/2013 10:20:08.875 PM racoon[1580]: p:1 t:1
10/26/2013 10:20:08.875 PM racoon[1580]: AES-CBC(7)
10/26/2013 10:20:08.875 PM racoon[1580]: SHA(2)
10/26/2013 10:20:08.875 PM racoon[1580]: 1024-bit MODP group(2)
10/26/2013 10:20:08.875 PM racoon[1580]: pre-shared key(1)
10/26/2013 10:20:08.875 PM racoon[1580]: 0
10/26/2013 10:20:08.875 PM racoon[1580]: 0
10/26/2013 10:20:08.876 PM racoon[1580]: lifetime = 3600
10/26/2013 10:20:08.876 PM racoon[1580]: lifebyte = 0
10/26/2013 10:20:08.876 PM racoon[1580]: encklen=256
10/26/2013 10:20:08.876 PM racoon[1580]: p:1 t:2
10/26/2013 10:20:08.876 PM racoon[1580]: AES-CBC(7)
10/26/2013 10:20:08.876 PM racoon[1580]: MD5(1)
10/26/2013 10:20:08.876 PM racoon[1580]: 1024-bit MODP group(2)
10/26/2013 10:20:08.876 PM racoon[1580]: pre-shared key(1)
10/26/2013 10:20:08.876 PM racoon[1580]: 0