There are several things that you can do to try to resolve the "unexpected error" message.......which is an ongoing bug in both Mavericks and Yosemite.
Power off VE Extreme (2) for a minute or two, then power it back up
Restart the Mac that you are using for the setup
Perform a hard reset on the AirPort Extreme that you are trying to set up by powering it on for a minute, then hold in the reset button on the back of the device for 7-8 seconds and release. Allow a full minute for the AirPort to restart to a slow, blinking amber light status.
Use a short, spare Ethernet cable.....any length will do to temporarily connect from one of the LAN <--> Ethernet ports on AExtreme (2) to the WAN "O" port on the AirPort Extreme that you are trying to set up
To get the setup "wizard" running, rather than open AirPort Utility manually, click on the WiFi icon at the top of the screen, then click on AirPort Extreme under the heading of New AirPort Base Station....like the example below:
If the AirPort Extreme is configured to "extend using Ethernet" correctly, then once the setup is done, you can disconnect the AirPort Extreme from the short Ethernet cable and install it back on the network using the Ethernet over co-ax devices.
If you still continue to see the "unexpected error" message, then we'll have to try changing some settings on the Mac to try to work around the bug that is producing the error.