I get the same thing, but only if I enter a "Group Name". Without a group name, I get timeouts during Phase 1 that I'm still trying to diagnose, but with a group name I get the same generic error message you do.
If I capture the generated config file and try and manually invoke racoon with it, I get the further information from racoon as such:
2010-04-02 15:39:29: [5778] INFO:
*** racoon started: pid=5778 started by: 627
2010-04-02 15:39:29: [5778] INFO: @(#) racoon / IPsec-tools
2010-04-02 15:39:29: [5778] INFO: @(#)This product linked OpenSSL 0.9.8l 5 Nov 2009 (
http://www.openssl.org/)
2010-04-02 15:39:29: [5778] INFO: Reading configuration from "/etc/racoon/racoon.conf"
2010-04-02 15:39:29: [5778] WARNING: /var/run/racoon/{VPN-IP-hidden}.conf:17: "support_mip6" it is obsoleted. use "support_proxy".
2010-04-02 15:39:29: [5778] ERROR: /var/run/racoon/{VPN-IP-hidden}.conf:101: "}" DH group must be equal in all proposals when aggressive mode is used.
2010-04-02 15:39:29: [5778] ERROR: fatal parse failure (1 errors)
racoon: failed to parse configuration file.
Apr 2 15:39:29 Thoth racoon[5778]: Configuration Parse Error. (cfparse: yyparse erred, filename /etc/racoon/racoon.conf). (failure: fatal parse failure)
Looks like it's time to open a bug report with Apple directly. (Which I'm about to do.)