It could be a 'difficult' form of Matroska file. FYI Matroska is just a container like MOV or AVI, and some mkv have the video encoded with Real Video Codec (RV9 RV10). Perian+QT can't decode this, but VLC will play .mkv files just fine on OS X, as long as they don't use H.264!
It's a bit of a lottery until Matroska decide to support the Apple platform. I have found this:
http://mkv4mac.free.fr/index.php?lang=en&p=lec
which may help.
But there may be another workaround: 1. Get Info the mkv file
2. Change to extension to wmv. Save as wmv and that could work better.