Outlook Setup for iCloud keeps saying my password failed

My AppleID password is obviously entered correctly because it opens the iCloud control panel just fine. When I try to turn on sync for outlook it starts, but then fails saying my username or password is incorrect. This is ridiculous, I simply want my contacts to sync with Outlook, this should not be that hard.