ilearn.jr, I think you've figured it out, but here's a little more information.
Basically, the iCloud API provides a fairly rich information conduit. Amongst the information passed between iCloud systems are custom fields, flag colors, and more.
The IMAP interface is not quite so rich. It has limitations that basically mean there is no "slot" into which a color can be sent, so disparate systems using IMAP as a conduit are out of luck. Some systems (e.g.: Microsoft Outlook) has "solved" this by attaching a big, nasty, proprietary, binary (well, hexadecimal actually) blob of data to the "Notes" field of contacts. This is great if you use Outlook -- and they stuff all kinds of things in there, including pictures, metadata, and more -- but unfortunately it makes a simple contact record (vCard) Really, Really Big, and also for anyone NOT using Outlook, you get this weird, big, nasty, proprietary, binary attachment that you can't do anything with. Bleah.
But I digress.
So the short answer is: No, IMAP systems cannot usually map things like colors and custom fields, but iCloud systems can, because it's a different and more customizable interface.
There may be a ray of hope, though. While tinkering with this, I did notice that colors seem to be pretty "sticky." So, I theorize (and lightly tested to prove, but haven't used) that if you get all the colors to "line up" on each side of IMAP connection, they'll stay lined up.
What I mean by this: I wanted my appointment calendar to be green. I connected iCal to Google Calendar, where it was green, but iCal made it blue. So I went back to Google and changed it to blue, and so far both calendars have remained blue. Granted, I couldn't get them both green (you get what you get, on the iCal side, I think), but at least they are the same color.
Not sure if there's a way to make that work for flags or not.
