If your VOB files came off a DVD, then you can play them through DVD Player by starting the app and then choosing File->Open DVD Media (or command-O) and selecting the VIDEO_TS folder.
If the files are not encrypted (ie didn't come off a commercial DVD), then you can also change the extension to .MP2; you might get some loving that way, but I haven't tried it in a while.
Just note that ffmpegX is a movie file converter. While it contains a player, that's not it's primary purpose.
The fact that a movie file is AVI doesn't say anything about which codec was used to compress it (AVI is like a container around the actual movie data). You have to have the right codec to play the file (usually you can download one for Quicktime, but some are proprietary and the owners want money). VLC usually does a good job when Quicktime fails, but although there are many VLC lovers out there, I've found its performance a little sketchy at times.