how do i play .mkv files in quicktime

Try converting the video. There is Smart Converter free from the Mac App Store. There are other converters too.

QuickTime itself has never supported opening MKV files let alone playing them. Perian an add-on for QuickTime does allow opening and playing MKV files although I don't know if DTS works this way.

Regardless there are some major limitations with Perian -

1. Firstly it is discontinued and no longer supported although you can still download it
2. Secondly it does not work in QuickTime Player X, it only works in QuickTime Player 7 (it is still possible to download and install QuickTime Player 7 here QuickTime Player 7 for Mac OS X v10.6.3 or later)
3. Thirdly even if it works due to the differences between QuickTime and MKV, it has to read the entire MKV file before it is properly available to play, you can start playing before it is fully loaded but you cannot fast forward beyond what has been loaded)

You are probably better off using VLC to play the MKV files.

encode the mov file using Handbrake. QT will play m4v or mp4 files.

Try something along the lines listed here http://www.mactricksandtips.com/2013/05/selecting-the-audio-output-within-vlc.ht ml the name of your device may be different but you probably still want to be looking for a Digital Encoded choice.

Also check the VLC preferences for Audio settings and see if options there help. In theory it is supposed to be much easier in VLC to do surround-sound that QuickTime. QuickTime requires messing about with the Audio MIDI Setup utility.