distorted screen recording when zooming in

Since the upgrade to High Sierra the screen recording by QuickTime Player is distorted when the screen is zoomed in (via the Accessibility setting in the preferences of Mac OS X). See the screenshot. Do you have the problem, too?

If I understand your workflow correctly (i.e., applying a "full screen" zoom effect while making a screen recording), then I get the same result. However, since I don't use this workflow, I can't say if this issue is something new under High Sierra or the normal screen capture operation. If it's different than screen capture renderings under previous MacOS versions, then I would likely consider this a bug to be reported to Apple. All I can really do is at this point is describe what appears to be happening during screen recording and playback.

Each application of the "Accessibility Zoom In" command in the "Full Screen" mode appears to select a fixed percentage of the current 0,0 coordinate (upper-right-hand corner) anchored display area for "zooming." During recording, this sub-area is scaled to fill the entire screen display but is composited at its original resolution and display position to the video track on the layer above the previous video layer. During playback, the entire video frame area is displayed with all layers visible—creating what might be called a compositing distortion. Zooming out during recording deselects the current "zoom" region and removes its associated composite layer from the screen recording.

Not sure what affect you are trying to create here. If the application of a zoom affect to a specific area of the display, you might try using the "Picture-in-picture" mode to turn the entire screen into a "Call Out" area and then focus the center of the zoom affect on the mouse cursor to control what area of the screen is zoomed in and out when using the accessibility commands. Not sure if this would be a viable workaround for you or not. Here is the URL for a quickie HEVC test file of this workaround: http://downloads.walker4.me/Downloads_files/ZoomTest.mov

I suspect this may be another High Sierra related issue/oversight like lack of QTX media player AVCHD/MTS and "hev1" FourCC support.

I find the same problem like your case. Do you find the solution yet?