So, I was finally able to get it work, mostly.
I actually went as far as to buy a Sonnet eGFX 550 box and RX580 (same as https://developer.apple.com/development-kit/external-graphics/) to test it all. First I tested RX580 in a Win10 PC and yes indeed, I was finally seeing 5K@60Hz on my monitor!
Then I moved on to test the eGFX box, but unfortunately the results were still the same as previously with the Moshi cable. Even though the GPU is definitely capable of 5K@60Hz, macOS still does not know about this.
I enrolled in the beta program to try out the 10.13.4 public beta, but it doesn't show up as an available update. 😟 I have found that other people are having similar issues, and the solution proposed is to reinstall High Sierra from Recovery mode — well, I'm not up to doing this right now.
Then I decided to try something else and actually fit into the DP1.2 bandwidth.
I disabled SIP, then created a number of custom resolutions in SwitchResX which went like 5120×2880@30Hz, …31Hz, …32Hz and so on. I rebooted to try them out… and some actually worked! Most were listed as incompatible, but everything at 38Hz and below does work.
I disconnected the eGFX and connected directly with the Moshi 5K cable. IT WORKED AGAIN!
Currently I'm running at native 5120×2880@38Hz, in glorious 2x retina, on macOS 10.13.3 via a single cable.
Image quality is very good and I'm pretty happy.
I would be happier if I had 60Hz though, but I guess it's now up to Apple and Iiyama to figure this all out. I hope I will be able to use the eGFX box for this later when 10.13.4 is out.
I have also noticed that the monitor is wrongly detected as PL2779QQ by default (as a non-retina monitor they released several years ago). Maybe it has something to do with it reporting wrong capabilities.
Anyway, I hope someone at Apple will notice this thread. I've asked them on Twitter, still waiting.