Cannot Get Mail: The user name or password for "imap.gmail.com" is incorrect

I have a user account to Google that doesn't end in @gmail.com as I have chosen not to use gmail for this account deliberately.

When adding this Google account to iPhone, I see that iOS is trying to authenticate against imap.gmail.com. This attempt fails as there's no mailbox there for this account.

Google however still allows people to manage your contacts without having a gmail account (different service).

Is there a way to add non-gmail.com account to the iPhone, so contacts can get synced up?

Thanks,

Miklos