Movies purchased from the iTunes Store use DRM (copy protection), so you cannot convert them.
It continues to tell me that it cannot sync because it is high quality SD video and is not compatible with the iPod.
Find the movie in your iTunes Movies library, right-click it, and Get Info. On the Info window's File tab, look at dimensions (the movie's screen resolution). What does it say there for this "high-quality SD" movie format? According the 3rd gen iPod nano's tech specs, it can play videos up to 640x480 (like most video-playing iPods that are not iPod touch).
Another user here pointed out a setting in iTunes preferences Store tab. It is a checkbox for Download high-quality SD videos. What you can try is to uncheck that setting. That movie is not available as a rental (yet), so I assume you purchased it (you can therefore download it again). If so, Delete the movie from your iTunes Movies library. Then, go to your Purchased screen in iTunes
Download your past purchases - Apple Support
Set it to show Movies and Not in My Library. Find the movie. At the bottom right corner, there's a checkbox for Download HD When Available; it may not matter if you bought the movie in SD (not HD) format, but uncheck that setting. Click the cloud with down-arrow symbol for movie to download it again. See if it now syncs to your iPod.
I don't know if this works, and if Apple has a separate "lower quality" SD movie file that downloads if you uncheck that Download high-quality SD videos setting in iTunes preferences. I'd be curious to know...
Also, if you don't mind, after it downloads again, check the movie's dimensions (on its Info window File tab) again. Is it different from before?
