icloud and non icloud users can't see invitations

I am using icloud and have turned off all of the ON MY MAC calendars as that was causing duplicates and a host of syncing issues when I did not select the right calendar.

It seems that when a non icloud user sends me an invite (mac user) that they do not appear, even as an email. Vice versa is also happening when I send a calendar invite to a non icloud user, they do not see my invitation.

What's the solution.