iTunes is set up to play videos in iTunes. It does not have an option to choose an external player. If you want to use an external player, you can control-click on the video, choose "Show song File", control-click on the file, and then choose VLC Player. Or, you can change the default app of the videos by highlighting a video in Finder, pressing Command-I to get the Get Info Box, changing "Open With" to VLC Player and clicking on "Change All". now you can go to the file in the Finder, double click on it, and it will open in VLC Player. Note that any video purchased through the iTunes Music Store can only be played in iTunes because they are copy-protected files.