can't play iso movies using vlc-el capitan and safari

ISO is not a video format.

What format is the video file (possibly) in the ISO you are trying to watch?

is it a proprietary disk?

is it a video disk? is it a blu-ray?

VLC is a standalone application, it does not "run" inside Safari.

An ISO file is generally a copy of an original disc, it could be a DVD disc, a Blu-Ray disc, or a computer data/program disc.

Assuming it contains either a DVD movie or a Blu-Ray movie you first need to 'mount' the ISO as a virtual disc, this is simply done by double-clicking on the ISO file. You can then tell VLC to look in the mounted disc and if it is a DVD or Blu-Ray play its contents.

Note: VLC does not itself have the ability to defeat any DRM protection on a DVD or Blu-Ray disc.

Note: VLC does not support playing the Blu-Ray disc menu, it does support playing a DVD disc menu. For a Blu-Ray it will let you jump straight to the main movie title and play it.

Note: Because sadly Apple do not believe in high quality audio it is impossible in any OS X program to play the highest quality audio formats as included on a Blu-Ray movie disc, therefore at best the audio will be equivalent to DVD level. Also sadly Apple still have not convinced the luddites in the music industry to allow selling lossless audio via the iTunes Store. With regards to Blu-Ray audio even Linux is ahead of OS X and Windows is way ahead.