How can I scale my external screen a bit up from 4K to around 5-6K resolution?

I haven't checked the specifications or user guide for your monitor yet, but there is one way that you might be able to use more scaling modes with it.

Does your monitor have the ability to accept two 3840x2160 pixel signals – one for the left hand of the screen; the other for the right hand of the screen – and display them side by side? If it does, you could run two video cables to the monitor. If it worked, the Mac would see the monitor as two external displays, each of which had a resolution of 3840x2160 pixels.

To make them look like one monitor (from your standpoint), you would run the two "monitors" in extended desktop mode with the same left-right arrangement as the actual halves of the screen. Based on the resolution choices my M1 Max Mac Studio offers me for my 27" UHD 4K monitor, the M4 Pro MacBook Pro might be willing to run each of the monitors at settings like these.

Selecting Retina "like 2560x1440" for both of the "monitors" would thus effectively get you "like 5120x1440" mode for the entire physical monitor.

You said that you have a M4 Pro MacBook Pro.

The Technical Specifications for MacBook Pros with M4 Pro chips read, in part:

----------

Simultaneously supports full native resolution on the built-in display at 1 billion colors and:

• Up to two external displays with up to 6K resolution at 60Hz over Thunderbolt, or one external display with up to 6K resolution at 60Hz over Thunderbolt and one external display with up to 4K resolution at 144Hz over HDMI
• One external display supported at 8K resolution at 60Hz or one external display at 4K resolution at 240Hz over HDMI

----------

Use an external display with your MacBook Pro - Apple Support describes the same capabilities.

The Apple Pro Display XDR – a 6K monitor – has a resolution of 6016 x 3384 pixels. Your Odyssey G5NC monitor, with its resolution of 7680x2160 pixels, has the horizontal resolution of an 8K monitor, even though it only has the vertical resolution of a UHD 4K monitor. (In effect, it is like two UHD 4K monitors joined side to side.)

So you may already be up at the point where you cannot make a USB-C (DP) or Thunderbolt connection, and must make a HDMI connection, and must limit yourself to a single external monitor.

Then you want to use a Retina scaling mode with greater than 3840x1080 pixels, on top of that. In Retina scaling modes, the internal drawing canvas has twice the number of pixels in each direction as the nominal "UI looks like" a.k.a. Displays Settings "resolution".

You would be asking the M4 Pro chip to work with an internal canvas with a horizontal resolution greater than 8K. Apple does not completely document how the resolution limits in Technical Specifications apply to Retina scaling modes. But it could be that the chip simply can't support an internal drawing canvas that has the same horizontal resolution as two 5K displays set side by side.

Procisz wrote:

So you want to say that this flagship gaming monitor by Samsung isn't compatible with Apple devices?

It appears that this is a 57" monitor with a resolution of 7680x2160 pixels, and a maximum refresh rate of 240 Hz. If you limit yourself to one external display, and you connect that display using HDMI, your Mac can support either 8K @ 60 Hz or 4K @ 240 Hz. There is no promise of supporting 8K resolution and 240 Hz at the same time.

Yet despite this, there is some compatibility between your Mac and that monitor, as evidenced by the fact that you are getting a signal. You just aren't getting all of the options that you were hoping to get.

As long as a display has more than one input and supports Picture-by-Picture (cousin of Picture-in-Picture), you can run it with two cables, one for left half and one for Right half. That display says it supports picture-By-Picture, and it features:

3x HDMI

1X DisplayPort

on the Mac, each half-display can be arranged back together in the Arrangement pane. in the display, by enabling Picture-by-Picture, using on-screen-configuration settings.

Any interface can normally be used, but at full resolution you will have two 4K displays side-by-side. If attempting to use USB-C adapters, these will be limited to 60 Hz using DisplayPort.

You can only get 60 Hz with an HDMI adapter using HDMI 2.0, which requires colorspace compromise to reach 60 Hz when starting from USB-C. Stand-alone Adapters that claim ThunderBolt to HDMI 2.1 are generally unfeasible, and will get burn-your-skin off HOT until they fail unreasonably quickly.

ThunderBolt to DisplayPort typically requires so much power it is only featured in Docking stations.

That would depending entirely on the external display’s abilities and compatibility with Apple hardware. Please contact your external display’s customer support team.

You told lkrupp

"I'm sorry, I'm pretty sure that it is not the display.

This is the monitor: https://www.samsung.com/hu/monitors/gaming/odyssey-neo-g9-g95nc-57-inch-240hz-curved-dual-uhd-ls57cg952nuxen/

So you want to say that this flagship gaming monitor by Samsung isn't compatible with Apple devices?"

As if, if you go shopping for a monitor with extreme or unusual specifications, that eliminates the need to do any research on how well its might work with other devices. That's not how things work. It is precisely when you are shopping for monitors that have very high resolutions or refresh rates that you need to pay the most attention as to how well they will work with other devices.

Perhaps you should have investigated before purchasing instead of making assumptions, at least for the resolutions you are trying to set on Apple hardware.

Users who have attempted the two-cable connection method report that its works great without any discernible limitations. Specifically, No seams, and no scrolling anomalies.