As I noted in my previous comment, third party utilities are available to convert any video to a format that is fully compatible with a given Apple product. In my case, I have a library of over 200 movie files which I converted using iMedia Converter and none of them have presented problems on any of my Apple devices including the Apple TV 4.
You don't know what the issue is until you identify what differentiates the files that work on your Apple TV from those that don't.
"MPEG-4" is not sufficient information. A video file is comprised of a video track and one or more audio tracks, each of which is encoded with a particular codec, resolution, bitrate, etc., and then the whole thing is enclosed in a particular type of wrapper. TVOS may have an issue with any number of those details in the files with which you are having issues.
This explanation illustrates why it makes no sense for Apple to attempt to support more video file types: the effort would be a monumental waste of resources with limited benefit, especially in the case of inferior and outdated codecs (e.g., WMV, FLV, DIVX, AC3, etc.)