ProRes video to H264 without losing so much color?

You shot the in-camera HDR format HLG, but it's unclear if you have an HDR monitor to grade that or if you intend to distribute in HDR. A mandatory part of any HDR format is 10-bit depth or more, and ideally 4:2:2, but you are trying to export that as 8-bit 4:2:0.

You also put that on a PQ project, and PQ has less SDR backward compatibility than HLG.

If you put the HDR HLG material in a wide-gamut library, on a Rec 2020 HLG project timeline, then export using this FCP preset, it will look well on both HDR and SDR playback: File > Export File > Settings > Format: Computer, Video Codec: HEVC (10-bit, HLG, Dolby Vision 8.4). It can be uploaded to YouTube and after it finishes encoding, it will look good on streaming playback to HDR-capable devices and also on SDR devices.

If you don't have an HDR monitor (e.g, 27-inch Studio Display XDR, 14/16-inch Apple Silicon MacBook Pro), you can shoot iPhone videos in Rec.709 or Apple Log, then grade it for SDR. If Apple Log, it retains HDR potential so it can later be regraded for HDR. So that is a major advantage of shooting in a 10-bit log format, even if you currently only need SDR.

If you only need to grade your existing HDR HLG material for SDR, follow these steps. It should not generally be necessary to use HDR tools. The purpose of the auto color conform is to eliminate the labor-intensive trial-and-error methods.

• Import to an SDR library
• Create an SDR Rec.709 project
• Put the HLG material in the project
• In FCP Settings > General, enable HDR: Automatic Color Conform
• In the clip inspector's video tab, enable Color Conform > Automatic
• If you have an Apple monitor with reference presets (e.g, 2022 Apple Studio Display), you can select the HDTV BT.709-BT.1886 reference preset. This will ensure it uses BT.709 standards.

If you have a third-party monitor, it's important to know what kind of ICC display profile is being used. That is in System Settings > Displays.

Suggestions:

Try exporting directly from FCP using these presets: File > Share > Export > Settings > Video Codec : HEVC 10-bit.

Then play that in QuickTime Player. Does it look OK? If not, when playing the file in QuickTime Player, do CMD+I (Get Info). Tell us the values in General *and* Video Details. See attached example.

Provide the CMD+I info for your (1) original ProRes 422 file, (2) exported 8-bit H.264 video from Compressor (3) 10-bit HEVC file exported using above settings.

Starting with macOS Monterey, there is an automatic OCR feature called Live Text that can save you typing and avoid posting graphic screenshots. Just take a screencap of the QuickTime Player CMD+I panel using SHIFT+CMD+4 and drag/drop over the dialog, then in Finder select the screencap file and press spacebar (QuickLook), then drag/drop your mouse over the text in the image. It will automatically copy that to the clipboard, and you can paste that into a post.

I’ve done hundreds, maybe thousands, of exports like this and have never seen this kind of signal degradation. However, mine have been Full HD, 8-bit ProRes files, not 4K 10-bit. In the type of work I do, I don’t need the small improvement that 10-bit provides.

There are quality settings on an H.264 export, and I always set mine to the highest quality settings. Is there a chance that you may have missed this and it was set to a low quality setting for your export?

More questions:

• Are you down-sizing the frame size to something smaller than 4K? If so, to what size?
• Are you keeping the same frame rate as the original ProRes files?

We’ll figure it out…

To review, you have a third-party SDR monitor, shot HDR HLG iPhone material, imported that to an FCP wide gamut HDR library, you created an HDR PQ project, added the HDR HLG iPhone clip to that project, made various grading changes including applying the HDR Tools effect, exported as 8-bit 4:2:0 Rec.709 using Compressor, and now wonder if you can salvage the chages to date and switch to an SDR Rec.709 project in the same wide gamut library?

I think yes, but I can't predict how much of your current grading work can be salvaged. I did some quick tests with similar material that implied the below steps might be possible. However I'd suggest duplicating your current project as a backup, and even shutting down FCP and duplicating the library in Finder.

Before you do that, delete all cache files to reduce the library size:

• In FCP, select library in left sidebar
• File > Delete Generated Library Files > Delete Render Files > All

Shut down FCP, locate library in Finder, right-click, duplicate. If it's on an APFS drive, this will take little space since APFS uses block-level file cloning. IOW it shows you the duplicated file but there is really only a single underlying file. As changes are made to either library, they will gradually diverge and start taking up additional space, but the original duplicate will not consume appreciable physical space.

Once you have those backups, you can try the following:

• Launch FCP and open your wide-gamut HDR library
• Open your HDR PQ project, select all clips with CMD+A then copy with CTRL+C
• Create a new Rec.709 SDR project
• Paste in the clips
• For each clip, select it in the timeline and verify in Inspector that HDR Tools is off, and Color Conform is enabled and set to "Automatic." Repeat for all clips.

Export from FCP as H.264 8-bit (if that's what you want) and see how it looks.

Edited: I re-read your comments and understood.

Regardless of what the original video was like an exported H.264/H.265 should look identical when exported . . . obviously there will be some slight difference but it should be undetectable under normal viewing.