Playing 4K video on 5K Mac display (LG Ultrafine 5k or iMac 5K)

Can anyone suggest a player that can handle the full res 4K video from the default Mac OS High Sierra's "Default for Display" resolution?

Unfortunately there is no such animal. You can't have true 4K (or 5K) resolution using the "Default" mode setting. Either you view the media content at a screen resolution capable of supporting the native resolution of the video or you view the content using the "Default" setting. The choice is yours.

BACKGROUND: Basically, Apple's "Default" resolution option converts a device's display to "Retina" by doubling the number of pixels vertically and horizontally, meaning it has 4 times as many pixels as a non-Retina counterpart. If it did that and nothing else then there would be a problem. User interface elements like menus and icons would look tiny (as you complained). To compensate for this, Apple created what it calls the "HiDPI" mode, where each interface element is doubled in size vertically and horizontally (i.e., displays at 0.5X its resolution) and so appears at the same size as it would on a non-Retina display—thus halving the 5K screen to an effective 2.5K display as you also noted. (I.e., the effect of a Retina display is to make everything look crisper. Text in particular benefits from Retina—it looks smoother, with the curves on characters looking like curves instead of jagged steps.)

Since Apple apps tend to be context adaptive, apps like QTX limit the player window to the "effective" resolution of your video display or, in your case, a 2560x1440 display area. (I.e., if you check the "View" menu, you should find the "Fit to Screen" the only available viewing option while most third-party apps function in a similar manner whether or not other options are greyed out.) Thus, you must make the choice of seeing all screen objects in the 5K resolution mode with text and icons too small to really read/see or use the Retina/HiDPI (half-resolution) "Default" mode which scales the video to fit the available screen display area.

I thought the idea behind hi-dpi UI resolutions was to make UI text and icons big enough, but keep full res for media, like photos and videos. I am sure there are apps that do it correctly, e.g. Adobe Lightroom for images.

I admit I'm not an expert but I have yet to come across a display card that can simultaneously set up multiple arbitrary areas of different resolutions which seems to be a prerequisite for what you want. As to apps, not being a an Adobe Lightroom user, I cannot confirm nor refute your claim. However, if it has this capability, then I would assume that it automatically resets the Retina screen from the "Default" 2.5K resolution to 5K resolution and then applies doubling to all non-image/non-video display elements/objects—i.e., the opposite of what Apple currently does.

Limiting the video to “simulated” UI resolution is a bug.

Then you should file a "Bug" report regarding this issue. Unfortunately, I believe Apple will claim there is no bug because the software is currently performing as it is programmed to work using four 5K resolution pixels to create a single 2.5K resolution pixel with HDR shading and not as you think it should on a 1:1 pixel resolution basis. On the other hand, you could request an enhancement to Retina program routines allowing the user to operate in either the current "Default" Retina mode or to default to 5K (or 4K resolution depending on the system) the way you think it is supposed to work at each user's discretion. If so, good luck in your efforts.

It seems that the bug is actually the "current size" display of quicktime player on Mac OS. In retina resolution it reports the current "retina" simulated resolution instead of the full-screen video res. However, the video is supposed to play full 4K (assuming you have a 4K or better screen). VLC and other video players also play full res video.

4K videos play the same in VLC and QTX on my system. The only difference is in the resolution mode. In the "Default" setting four of the 4K video pixels at 5K resolution are used to generate a single display pixel with the 4K video then being scaled to a 2.5K display resolution with no discernible space between pixels. in the 5K scaled mode, the entire, unscaled video is displayed at its original 4K resolution covering 75% of my 5K display width instead of the entire scaled screen width as in the Default mode.

Frankly, I don't really see any difference between the "Default" Retina display at 2.5K resolution scaled to 5K and the 4K video display at 5K resolution scaled to fill the entire 5K screen width as seen below:

5K Mode

Default Mode

Unfortunately, I had to convert the PNG screen captures to JPG since the PNG were too large to upload and HEIC aren't upload supported yet.

Please see my previous comment. The video overlay works as expected on Mac OS, rendering full 4K resolution on displays that support it, the only confusing issue is that Quicktime player reports current video size wrong in inspector. The other players, like VLC and the new good one, IINA also use video overlay in full screen mode, so they can render full 4K res. For your information, Apple's own Final Cut X can also render video at 4K while UI is displayed at lower "retina" resolution.

QTX "Current Size" is based on the "Default" 2.5K screen resolution to which the 4K video is scaled in the QTX player window for display. Have yet to find a "Current Size" reading for the VLC video display. Both QTX and VLC indicate the source video to be 4K but only QTX provides Current and Encoded resolutions while QT7 provides Encoded, Normal, and Current resolution settings. Where are you finding the VLC "Current Size" reading to which you refer?

If you are so worried about this issue, why don't you set up two monitor screens with one set to the Default resolution and the other the 5K resolution and then simply move the video player window to the 5K display and leave the other display elements displayed on Default display. Am assuming this is possible even though I don't run a dual display system. Could wrong so maybe someone else should comment here on this capability.

Thanks for the heads-up on FCPX. Have yet to move on to it after FCP Studio was left unsupported by High Sierra mscOS update. A good reason to re-evaluate current software. As to IINA, it looked interesting so I downloaded it. Unfortunately, it also only reflects the encoded resolution like VLC—not the current display resolution. To verify this, simply expand the VLC or IINA video display window and recheck the inspector. You will note the display size did not change when you enlarged the video display area.