I didn't even think that the Retina display models were sold with only 4GB? What type of videos are you watching? Internet, stored on your hard drive, DVDs, etc.? Yes, your computer should be able to play any videos without choking but we need to know the source of the material before we can even attempt to diagnose the problem.
It shouldn't be freezing at all - it should just play smoothly. One thing - nothing to do with watching video - is that I would upgrade my RAM to at least 8GB. You can do it yourself - try www.crucial.com and use the System Scanner to see the correct RAM for your machine. If you've a 2012 non-Retina MBP, you can upgrade to 8-16GB at a very good price.
Now back to the video...
Which app are you using to play it? Quicktime player or maybe even VLC? I'd really recommend the latter. But the first thing that I'd like you to do, while the video is playing, is to open Activity Monitor (found in Applications>Utilities) and take a look at what may be hogging your CPU. Anything over 13% or so isn't using the processors efficiently. I'd also like you to check you System Memory and see if you've a lot of page outs and swaps. Please note both whilst playing the video. Something is either hogging your CPU or your RAM and we need to find out what it is.
Post back - I'll be here all night...