You need to meet us halfway here. What do you want from us? You asked if this was expected behaviour and asked for workarounds. We answered truthfully to the best of our knowledge. But that's not good enough?
On these forums, I regularly see people complain about network performance on macOS. These questions are not like your question. They always follow a very precise pattern. They do not deviate. In each case, macOS was the absolute epitome of networking power and speed, besting all other operating systems, right up until the release of macOS 26.1, or macOS 26.0, or macOS 15.7.1, or macOS 15.6, etc.
My answer is always the same. The last time I used macOS on a network was 2017. We had literally unlimited funds for hardware and support staff. And the support staff was uncommonly competent. macOS was an absolute dog. It would regularly lock up the system due to icon previews. Attempting to open a file on the server was guaranteed corruption. The only way to use it was to treat it like it was an FTP server. Copy a file locally, edit it on the hard drive, copy it back to the server. So I'm always skeptical that Apple somehow managed to overcome that only to break it all every year, and then apparently fix it every year.
I just so happen to have an Ubuntu server right here. I still have my old clip art package from like 20 years ago. Make I can actually use it for something. I copied all of the JPG images into one folder of 67k images. Then I opened that on macOS Finder (Sequoia, actually). It locked up the operating system requiring a reboot.
So then I started smaller and added more files:
3000 files - Linux ls 0.086s, macOS smb ls 35s - 1.2s, Ubuntu file browser ~1.0s, macOS smb Finder 3s
10000 files - Linux ls 0.279s, macOS smb ls ~112s - 3.2s, Ubuntu file browser ~3.0s, macOS smb Finder 8s
20000 files - Linux ls 0.570s, macOS smb ls ~180s - 6s, Ubuntu file browser ~4s, macOS smb Finder 2s
Note that all Linux and Ubuntu values are local disk access, not samba. My experiences with similar problems on Linux years ago were on a large NFS network.
Note: After 3000 files, I made a point to turn off icon previews in Finder > View > Show View options for this folder. Before I did this, I noticed that it behaved differently than I remembered. It didn't seem to load all icon previews at once. Instead, it would load them on demand, as I scrolled down. It was a poor user experience with heavy lag. Turning off icon previews eliminated the lag.
The command line interface behaved very strangely. The first time I got a listing in Terminal, it was horribly slow. But subsequent attempts were much faster.
Finder seems to rely very heavily on caches. After putting 20,000 files into this folder, Finder responded instantly, but said there were only 10,000 files there. It was only after I let Terminal ls grind out for 3 minutes did Finder show all 20,000 files.
You might want to try creating a new folder on your server with only a few files. Open that with Finder and turn off icon previews. Then move the rest of those 20,000 files there.
After all of this, I opened a fresh connection to the server with my Tahoe test machine and loaded that directory in the Finder. It took about 15 seconds to load. It was a little bit laggy. It took about 3 attempts to eject the server because it claimed that "Finder was using it". (It wasn't).
Of course, there are no icon previews. I can still do QuickLook previews with the spacebar. If my experience from 2017 is any guide, that will destabilize the system eventually.
I'm afraid that's as good as it gets. If that isn't good enough, then you'll just have to use Windows.
PS: Most tests conducted with a 2021 16" MacBook Pro with 32 GB RAM on a 1,200 mbps wireless network. The Tahoe machine is a 2023 M2 MacBook Air with 16 GB RAM on the same network. Linux machine is an ancient Acer AMD desktop. But it does have a wired 10 Gbit connection to the network.