I had a mp4 file which is AAC(LC)+AVC(High@L5, CABAC/11RefFrames, 1280*800 resolution), my QuickTime (Version 10.4 (855)) can play it normally on Mac 10.11.4. But when try to export this mp4 file with QT into 720P or 480P, it failed and reported the error message "Cannot Decode The media data could not be decoded. It may be damaged.".
QuickTime can play it well, that means this file can be decoded normally and successfully. But when exporting into 720P or 480P, that means that transcoding is must (1280*800 -> 1080*720 or 640*480). It's weird.
I agree with your logic here but remember that the Playback and Modernizer routines embedded in El Capitan are separate, independent structures. I suspect the Modernizer error traps are programmed differently than the Playback structure which may be the basic problem here. Correction of the error trip routines would probably require reprogramming by Apple. Please send feedback to Apple regarding this issue.
Is there any limitation of QuickTime but no any official introduction about it? How can I fix this problem?
Believe MPEG-4 AVC support is limited to at least High Profile @ Level 5.0 for QT players/devices—according to the Apple web site. (I.e., not sure if L5.1 is now supported by all players and devices or not and have had problems with L5.1 sample files in the past.) I personally am updating my old files (Main Profile, L3.1 anamorphic) to High Profile, L4.0 non-anamorphic use as a "universal" player/device format for my iTunes library content.
In any case, my immediate workaround recommendation would be to use the free HandBrake app to convert your file. However, be advised that your file is very highly compressed and will likely be larger than your current file if you encode at similar resolutions and try to maintain a similar level of quality. Also note that the aspect ratio of your listed file is 16:10 and a 1080x720 file has an aspect of 3:2 while a 640x480 file has a 4:3 aspect ratio. (I.e., a "fit within" 720p file would have dimensions of 1152x720 and a 480p file would be 768x480 pixels at a 16:10 aspect.)
