Tim,
I have the same issue. Although I couldn't get rid of the duplicate account, I was able to disable one of them from appearing in the Calendar app by doing this:
1) Opened the Calendar app
2) Went to the "Calendar" file menu and clicked on "Accounts".
3) Click on the exchange account and uncheck the "Calendars" (don't worry, we'll re-enable this later)
4) Close the Internet Accounts window and go back into Calendar. You'll see that you'll only have ONE instance of your Exchange calendar enabled and that one is the duplicate.
5) Go back to the "Calendar" file menu and select "Preferences".
6) Click on the "Accounts" button
7) You'll see the only enabled Calendar for your exchange, click on it and uncheck the "Enable this account" option.
8) Lastly, go back to where you were in Step 2 and 3 and re-enable the "Calendars" option.
I unchecked the "Calendars" option in step 3 rather than just going to step 5 is because I wanted to make sure that I was using the Calendar associated with the one from the System Preferences - Internet Accounts. Hope this helps you! I did this for both my iMac and MacBook Pro and days later still working!