Do I have to be connected to WiFi to watch/Listen to an item I downloaded from iTunesU?
Only if you are streaming the media. If it is download to the iPod wifi is not required.
This may help
iTunes U: Frequently asked questions
After reading this and the iTunes FAQ section you mentioned, and then trying it out, it seems that I can hear the lectures but not view the video. Is this what is meant by "only if you are streaming video?"
That makes no difference.
Are you sure the iTunes U media is not audio only?
Also. is it compatible with the iPod? Some are iPad only and while then may play audion on the iPod they wil not show video.
Retrieving data ...