I'm having the same issues.
With school starting next week, I'm trying to get the family organized.
Just sent invites for an event to my husband and 2 sons.
Husband received his email at work and accepted. (Although iCal still has a "?" by his name and I never received an email that he has accepted) As for the boys, they never receive an email at all.
The idea was wonderful at first but now it makes me want to go back to crayons and paper. :-P
I'm having this exact problem right now. I have 12 invitees and everyone who is having a problem not receiving the invites is either using GMail or as GMail based-email with a custom domain. I do have one GMail user who did receive the invitation and who replied.
Has anyone figured this out? Is it Apple or Google or something else?