I had these same issues. My shares mounted in macOS with no issue however, during file transfer, it would kill my whole network stack. I am talking the transfer would stall after 20 seconds to 2 minutes, then the entire network services on macOS would failover, then NAS share disconnects and transfer failed.
My setup was mounting to some Synology NAS shares (DSM 6 and 7), TrueNAS shares, and just regular old linux box shares.
All other client (non macOS) were rock solid, either via SMB or NFS. macOS clients, absolute garbage with SMB.
I tried all sorts of suggestions and configurations to /etc/nsmb.conf. I tried all manner of permissions and service configurations with DSM and exports. These SMB transfer fails are HEROIC failures with macOS.
What has finally worked for me is the absolute DUMBEST fix for an issue I have come across in my 23+ years of user/admin of such environments.
When authenticating to mount the shares, change one character of your user name to be a CAPITAL letter. For example, if you normally mount with a user name of:
nasuser1
Change the user name to be:
Nasuser1
As far as I can tell, adding a capital to any character in your user name has the same effect.
This 100% fixed my file transfer problems on macOS SMB mounts. All clients.
I do not know if this is an artifact of keychain drift over time, some odd formatting of saved credentials, whatever.
Just in case, I am pasting my nsmb.conf content here if it is useful to anyone. A few things are specific to my environment like forcing SMB 3 and disabling signing (I only mount SMB on shares I own in my own network so I am relatively confident in the authenticity). I absolutely HATE the idea of globally disabling such a security feature as packet signing, but alas, Apple has foisted this on us with poor implements of SMB...but I digress.
nsmb.conf
streams=yes
soft=yes
signing_required=no
dir_cache_max_cnt=0
dir_cache_max=0
dir_cache_off=yes
protocol_vers_map=4
port445=no_netbios
validate_neg_off=yes
notify_off=yes
mc_on=yes
mc_prefer_wired=yes
So, in conclusion:
I tried all the nsmb.conf optimizations I came across in hundreds of forum posts. They definitely improved the overall transfer speed when transfers were running, but the connection drops that killed my whole networking services on client were ultimately fixed by adding a capital letter in my username when authenticating.
No clue as to why, but for those that have tried everything else, what have you to lose?