May be not simply a problem with Apple's software (10.9.1 Mavericks) but (also) with the router – Cisco noticed something similar (on http://www.cisco.com/en/US/docs/security/vpn_client/cisco_vpn_client/vpn_client4 8/release/notes/48client.html):
If you use the VPN Client with a Digital Certificate and your Client sits behind a Cable/DSL router or some other NAT device, you might not be able to connect to your VPN Gateway device (that is, the VPN 3000 Concentrator). The problem is not with the VPN Client or the Gateway; it is with the Cable/DSL router. When the VPN Client uses a Digital Certificate, it sends the Certificate to the VPN Gateway. Most of the time, the packet with the Certificate is too big for a standard Ethernet frame (1500), so it is fragmented. Many Cable/DSL routers do not transmit fragmented packets, so the connection negotiation fails (IKE negotiation).
This problem might not occur if the Digital Certificate you are using is small enough, but this is only in rare cases. This fragmentation problem happens with the D-Link DI-704 and many other Cable/DSL routers on the market. We have been in contact with a few of these vendors to try to resolve the issue.
Testing with the VPN Client Release 3.1 indicates that VPN Client connections using Digital Certificates can be made using the following Cable/DSL routers with the following firmware: …
Compare also https://discussions.apple.com/message/19684739#19684739.
And finally compare also https://discussions.apple.com/message/11230155#11230155 finding that the certificate has to be placed in the system keychain, not the login one. (Did not solve it for me.)