The footage outputs as AVI file format
AVI isn't a format; it's a container. The video and audio streams contained within could be made from any number of codecs ... perhaps the codec needed to play the file properly isn't installed on your Mac.
Try downloading the Perian package:
http://www.perian.org/
Afterward, see if the file will play properly in QuickTime Player or MPEG Streamclip. If so, your problem is solved. If not, you really need to start over and capture the VHS footage on your Mac. Or convert the footage on your Windows system so they will play on your Mac.
I've also tried to export to MPEG
If you want to edit in Final Cut, do NOT export to any MPEG format.
-DH