Videos won't play in photos.

I have MacBook Pro (13-inch, Late 2011). I have always had recording of my son since a baby, he's now 6. Despite the videos showing up in Photos, they won't play and I get the message "Photos cannot play this video because it uses an unsupported format".

This usually means that the referenced file contains third-party, proprietary, or "legacy" compression formats that are not "natively" supported by QTX structures embedded in post-Mountain Lion Mac operating systems which are now used by QTX, iTunes, Photos, etc. for playback. Such files can usually be transcoded to more modern compression formats like H.264/AAC or HEVC/AAC in MOV, MP4, or M4V file wrappers as desired by the user and/or required for native playback compatibility.

I have updated to High Sierra, have VCL and Quick Time installed. It will play some very old videos, but there are more it won't play than will.

Not sure what "It" refers to above. When it comes to native playback support, QTX has a "closed" configuration built into High Sierra which only supports a limited set of compression formats. On the other hand, VLC contains its own, built-in codecs supporting most common third-party, proprietary, and/or "legacy" compression formats and the QT7 Player is supported by an "open" codec configuration which the user can modify manually as needed for playback of specific compression formats.

If the videos are showing in Photo, with a length of time of the video showing, surely they are still there (concerned somehow they are gone)?

Until such time as you go through and check the files individually, my initial assumption would be that the files were imported and stored properly but have become playback incompatible over time as a result of macOS updates. All you need do is open the your Photos library and check the master video files to determine if they have a playback issue, are corrupted, are orphaned, or are missing.

Shall i try and roll back the operating system ?

I wouldn't. I'd just attempt to open the master files in an app like VLC. If they open, then check the media info to determine the compression formats used to create the files. If not already H.264/AAC, I'd probably convert them to H.264/AAC using HandBrake. If files are orphaned but still available, I'd attempt to restore the correct path to the target file. If the file container is corrupted but the data still intact, I attempt to copy the data to a new file container. However, if the file data is corrupted or the master file is missing, you are probably 'SOL' if you didn't keep back-up file and/or library copies.

I have attempted to export an original unmodified file, and i am informed the file doesn't exist. I guess from that, the files have been deleted (even though as you say some legacy exists) ?

Or it could simply indicate the file is "orphaned"—i.e., cannot be located within the library by the database directory. As previously indicated, I'd recommend you open the library and physically check the "Master" folder contents to see which images and videos are actually stored on your system. Will screen capture a "Quickie Tutorial" showing how to do this...

QuickTip-PhotosLibrary.mp4

You can Control-Click or Option-Click the above link to download the video for viewing locally on your system. Basically, use the context menu (Control-Click the "Photos Library.photoslibrary" file) to select the "Show Package Contents" option which will then open the library as a folder. If you then open the "Masters" sub-folder, you can directly access any and all of your original photographs and videos stored in a year-month-day folder hierarchy.

I've learnt a little from.the instructions and I will use the Master to make copies onto to CD as well as back up on to hard drive.

With the price of USB docking stations in the $20-$40 range and multi-TB internal SATA document drives selling for less than $100, most of the members of our user group have switched from optical discs to the use of SATA drives as "removable" storage media but, if you can afford it, RAID units are still probably a better long-term investment.

Devastating, but such is computing.

At least you know for sure now and can take steps in the future.