Album covers are normally present. You may have to sync a second time to get them if they weren't there before.
Another trick I use is to turn off iTunes Match on the iPhone (even though I do use it on my computer). Set Settings > Music > iTunes Match to "Off". That will force syncing with the computer.