That's an iPod touch 3rd gen, I assume... 🙂
According to the specs page for that iPod touch
http://support.apple.com/kb/SP570
it can play videos (in a supported format) that is up to 640x480. If the video you downloaded from the iTunes Store was "HD" (high def), it would be 720p or 1080p, which is higher than 640x480. That may be why you got that error message. In iTunes, the video's name in the library listing may have "HD" next to it.
If this was something like a TV show episode or movie, you should be able to find the same video in "SD" (standard def) format. If it was a purchase, I think you can download the SD version without paying again. When the SD version is downloaded, the video's name in the library listing may have an "HD-SD" next to it.
If this was some other type of HD content, like as a podcast or iTunes U, you may be able to convert it. Select the video in the iTunes window. In the Menu Bar, under Advanced, select Create iPod or iPhone Version. That version would be SD.
Once the SD version is there, try syncing it to the iPod touch again.