I did a bunch of debugging as I have the same issue, and annoyingly the upload of some images failed and the post was lost.
Anyway, after I hit submit I did some more digging and found the culprit, which wasn't in the original post.
The yellow errors are thrown when I changed cameras in Photo Booth from either my Logitech C920 (connected to the USB ports on the back of the display) or the laptop's Facetime camera to the LG 5K display's camera.
The repeating pairs of
DECODING ERROR -12904
RETURN ERROR -12904
tell us that the
decompressionOutputHandler
method on line 3374 in the
CMIO_Unit_Convertor_VideoToolboxDecompressor
C++ class is receiving an errant response that it can't decode, likely frame data which needs to evaluate to a set of numbers of particular format which constitute a valid basis for conversion by the decompressor into a frame as displayed by the output application (Photo Booth in my test case).
Unfortunately, that source file is part of the CoreMedia framework itself and is thus closed source, so I can't actually see exactly what it's doing, but I'm pretty certain the scenario outlined above is the likely case.
So now we know part of the what that is causing the symptoms, but we still don't have a root cause for what is causing that nor the why it is happening on this model only.
On that, I'm guessing that the Intel graphics hardware this model contains, the Intel Iris Plus, Device ID: 0x8a53 and Revision ID: 0x0007 (which perhaps the current 16" has as well, though both models there would likely hand off to the AMD GPU thus masking this bug) is generating some data which results in an array (which represents the numeric data set that a video frame is encapsulated in) boundary being overstepped so only 1 in several fall into line with the expected format (apologies to anyone on this thread who isn't a software developer, this is about as high level as I can make it; but think of a clock which needs to show 12 o'clock every time you look at at it, but each time you look away it increments the time by 1 hour; you only get a valid response once every 12 attempts), resulting in a large volume of dropped frames and juddering output.
Sadly, the only way I can test that out is with an eGPU for the OS to delegate the graphics offload to with a suitable application (I checked and it doesn't look like Photo Booth would do this, but I think zoom does if a real GPU is available).
So who needs to fix this? Well, that's tricky because I haven't conclusively proven what is at fault, but my first port of call would be Apple, to patch that method to handle the returned data that can't currently be processed, effectively fixing the broken array by reindexing it before returning the value (assuming my hypothesis above is correct).
I'll add my findings to the chorus with a bug ticket.
