Mac Calendar "Server responded with an error" message

Okay here's how I fixed it, this assumes you have no On Your Mac calendars, so back those up ahead. Also backup any Mail account settings like extra email address aliases. Try this at your own risk.

1. Close calendar
2. Go to activity monitor and end the process CalendarAgent
3. Delete your Google account in System Preferences
4. Go to your Google account and revoke MacOS permissions from your 3rd party app access. https://myaccount.google.com/permissions
5. Go to ~/Library and outright delete the Calendars folder
6. In ~/Library/Preferences delete com.apple.iCal.plist
7. Reboot your machine for good measure
8. On reboot re-add your Google account
9. Start-up Mail, Contacts, and Notes to get the resync started.
10. Calendar should startup like it's a fresh install but will sync.
11. I then used Caffeine to keep my Mac awake and left it to sync my massive Mail/Contacts/Calendar overnight (it was late but I had no errors so I figured it would take time).

After doing this, everything worked.