Apple Event: May 7th at 7 am PT

Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

ODY1980 Author

Level 1

9 points

Imac Pro - Crash when copy over 200GB on SMB !?

Hello, I have a really strange problem - since I upgrade to my new imac pro.

I really hoped for the 10.13.3 update - because it mentioned to fix SMB problems with crashes - but sadly it didn't help.

The problem I have is the following:

When I copy my VM Ware Folder (via 1Gbps ethernet cable) as a backup to a synology via SMB as soon as I hit around 200GB over the network the network of the imac stalls. No more receiving/sending - no more possibility to ping - even trying to disable and reenable the network doesn't work. No errors shown in any network utlity of bad packets etc. If I just wait then the mac just reboots itself without any error message and comes back.

What I tried so far:

1) Another imac Pro - same issue!
2) Even though the lockup happens on the imac (so it can't be solely related to the network itself) i tried to check if there is anything wrong in the network. Changed router, Change switch, Changed network cable etc. ! However with an old imac I can copy the file on the same line easily without any issues.

3) Tried different SMB settings 2.0 / 3.0 / disabled signing etc. - everything without any help.

For me its clear there is a severe driver issue with the new 10GB network port on the imac pros - but how can I truly identify the issue and report it to apple - so they take me serious without doing the normal troubleshooting...

If anyone has an idea, I would be more than happy to hear it 🙂

Like i said if i for example copy 50GB and then later 150GB - it will happen at 150GB - so it look like something is running full in a some kind of buffer at 200GB copy over network to smb.

Kind regards,
Oliver

macOS High Sierra (10.13.3), null

Posted on Jan 24, 2018 12:29 AM

Best reply

ODY1980 Author

Level 1

9 points

Posted on Jan 24, 2018 8:44 AM

Hey there - with your help I was finally able to find the issue.

Manually did not help me in the beginning because 1000T was correctly set by automatic. But automatic also sets the full duplex mode to:

full-duplex, flow-control, energy-efficient-ethernet

The problem here is the last option -> energy-efficient-ethernet <- this causes when using the SMB protocol and copying more than 200GB of data to the network to completly crash or stall the imac pro.

So all you need to do is changing it to manual and setting the full duplex to: full-duplex, flow-control

without this energy-efficient-ethernet - this also boosts your network speed by 2-3mb/sec.

I reported this to apple and hopefully they will make a fix with the automatic setting.

Kind regards,

Oliver

View in context

36 replies

ODY1980 Author

Level 1

9 points

Feb 3, 2018 2:26 PM in response to ODY1980

After a lot more investigation of this problem and due to the fact that it reappeared for me since I reproduced it for apple - it seems like I finally found the real culprit of this issue. After reinstalling the mac the problem was still persistent - I couldn't even do a time machine restore with the imac pro via my SMB backup due to this critical network issue - so I can imagine there are a lot of people out there and pulling their hairs off why their machines are crashing or why their time machines backups won't get restored over a SMB network...

Interestingly the energy-efficient-ethernet seems not to be the issue - it must be somehow related but the real issue is the new TCO (TCP segmentation offload) - a feature which offloads work from the CPU to the network card in order to safe processor power. However this feature was already be known at some point to cause issues on large file transfers.

I found this by coincidence when comparing why the external apple gigabit ethernet adapter didn't cause any issues at all but the internal in the imac pro does. So this ethernet option was the only difference! It may also explain why people with the sonnet 10gb adapter experience the same - pretty sure that this card also uses TCO !

So in order to disable that all you have to do is typing the following command into the terminal:

sudo sysctl -w net.inet.tcp.tso=0

This will immediately turn TCO off - no need to restart the network etc. ! However if you are doing a reboot you need to enter this command again. I don't know on how to make this automatic yet - since it requires sudo rights and can't be simply executed as a login script. But I think this will help apple track this down pretty easily and hopefully they will apply a fix for that soon enough.

Please try it out and report back here if it worked for you or not!

Feb 4, 2018 6:22 AM in response to ODY1980

I think this is it!!

My testing:

- changed tso-parameter to: 0

- mapping a SMB connection (to local NAS)

- copied around 400 GB without any stuck/disconnect or crash

- disconnected the smb share

- switch the tso-parameter back to default (1)
(no reboot between any step)

- reconnected by smb to the same machine and folder (NAS)

- started a new copy job (with the same files)
After around early 50GB the network connection was kicked (disconnected) and the machine is getting more unresponsive.
Done then a manual reboot to prevent corrupted files, before the machine get a kernel panic.

On my opinion this, the net.inet.tcp.tso parameter (or something at the network driver in bond with the tso 'thing') has to be the imac pro fault with network shares.
Good job ody1980 👍

Possible any other can confirm this?
At monday afternoon there's a planned call with Mike from the Apple Engeneering Department (In the meantime, had sended them a machine wide log after crash). If he's calling through again, i'll get him a hint on that.

Feb 5, 2018 1:28 PM in response to Wunderlicht

Yup! I'm finding the same kind of results! So it seems that net.inet.tcp.tso is the issue!

Honestly @Ody1980, I could hug you right now. I mean, this seems to have fixed my issues completely!

My 'replacement' iMac Pro is currently sitting at the local Apple Store. I wonder if I'll even bother going to pick it up now, because honestly, I don't want to go through the hassle of swapping out a machine that doesn't NEED replacing. I need to get Apple to OK this though, because they wanted to see my machine.

As for offloading processing to the network card? Well, I really don't think this 10-core is going to suffer much of a performance drop...

ODY1980 Author

Level 1

9 points

Feb 5, 2018 1:26 PM in response to Nikki Gordon-Bloomfield

I am really happy to see that it indeed looks like the culprit 🙂 Apple is already informed about that so I would say they will soon be able to address it. Curious now on how they will solve it - hopefully they can do a driver update and fix the network card issue with TSO - because TSO is not a bad thing actually!

Feb 6, 2018 5:41 AM in response to ODY1980

We did a transfer last night with this command and it seems to have worked! No crashes! Unfortunately though, it has cut our Read/Write speeds in half. From 700/400MBps to 300/250MBps :-(

Anyone else having this reduced speed issue?

ODY1980 Author

Level 1

9 points

Feb 6, 2018 5:45 AM in response to IndianaPwns

Well I don't have a 10GB network, but for a normal gigabit ethernet - there shouldn't be any reduced network speed.

I am even getting 2-3MB/sec more with this setting from 113Mb/sec to 116Mb/sec - which is the max you can get over a 1Gbps network.

You sure that this is related to this option? You can easily change this setting on the fly in the terminal from 0 back to 1 and you should immediately see the difference in network speed. There is no need to reconnect to the SMB nor restarting the network. TSO is applied/disabled instant. Thus you could easily track down if its really this option what is causing this.

Kind regards,

Oliver

Feb 6, 2018 2:35 PM in response to ODY1980

So, it seems that this is the golden bullet for this particular problem!

Yesterday, my replacement iMac Pro arrived (I was asked by Apple to return the first one even though we thought there was a solution at that point 'just in case') and I took it to my studio last night, entering the command to turn of TCP.TSO

This morning, I got a call from Apple, and we went through the steps required with the replacement iMac Pro to reproduce the issue... and bam... as soon as TCP.TSO was turned on, back came the problem!

Since ending the call, I've turned off TCP.TSO again, and the machine has worked flawlessly. So here's hoping the Apple Engineers make a patch ASAP to fix this annoying issue!

Nikki.

P.S. Now the thing isn't crashing, I'm really enjoying how quickly this 10-core 128GB machine slices through FCPX footage like butter. Oh yeah. Mmm. Butter...

Feb 8, 2018 8:44 AM in response to ODY1980

Definitely related to this issue! Roger that on changing things on the fly, but obviously not optimal.

Is there any "reason" you or anyone can think of that would cause things to be slower with TCP.TSO off?

Feb 13, 2018 8:59 AM in response to IndianaPwns

They got my full crash-log and kernel panic report.
We have to wait for one of the next updates, he says. Don't think that Apple is calling -me- to offer a pre-fix or for the "fixed-it message". I am only a ordinary customer... 🙂

The two calls from Apple were suprise and effort enough. Never experienced this with any other big company that they call 'the little man' and overseas talk!
Now they have work to reproduce and fix this cause. Hoping at the next coming High Sierra update. Fingers crossed.

ODY1980 Author

Level 1

9 points

Feb 13, 2018 12:18 PM in response to IndianaPwns

Same here and I have to say it was really nice to see that you get pretty quick someone of a higher level to talk to without too much hassle - especially after I told them that they had a serious issue with the networking adapter in the new imac pro 🙂

Now they have all infos they need to fix it and I am pretty sure they will do it soon enough. I mean this is a serious issue affecting many customers of imac pros - especially since even a simple restore on the imac pro from an SMB drive will cause serious troubles.

I myself invested more than 2 weeks to find the fix and I was already thinking my imac or my network is at fault 🙂

So let's see when they will provide us with a solution and hope they can calm down any customer who is facing this quick. But overall for such a big company it was great to see that their support was really trying to help - also had an oversea calls etc. 🙂

Kind regards,

Oliver

Feb 18, 2018 9:26 PM in response to Nikki Gordon-Bloomfield

I got a replacement iMac Pro and so far it hasn't given me a single kernel panic. I have been using the "sudo sysctl -w net.inet.tcp.tso=0" command from earlier in the thread on every boot, so between the new computer and that command, I am panic-free.

But I have experienced a couple of instances where my networking will just stop entirely and only be remedied by a reboot. One just occurred a couple minutes ago, and I tried pinging my router in Terminal just to see what it said, and it gave me this error:

"ping: sendto: No buffer space available"

Do you know whether this occurred at all when you had your network issues? And has this occurred at all since you got your replacement machine & started using "sudo sysctl -w net.inet.tcp.tso=0" command?

Feb 18, 2018 11:23 PM in response to mustgroove

Yes, this is exact the error. When you wait any longer the iMac will reboot by itself simply by clicking 'empty' on the desktop.
Exact so it happend right before a reboot by tgeh machine itself: First network disconnects, slow(er) response, then this message when sending a ping, then the machine reboots itself.
I think you still have the fault in your new machine. We had to wait for a software fix.

ODY1980 Author

Level 1

9 points

Feb 18, 2018 11:26 PM in response to mustgroove

Yep that's why a new iMac Pro won't help you 🙂

The issue is solely related to the network adapter and this TSO. But if you set it to 0, you should not get those reboots and your network should not start to stop functioning.

Kind regards,
Oliver

Feb 19, 2018 8:33 AM in response to ODY1980

It's possible my situation might be slightly different though:

1) I'm not getting spontaneous reboots or panics when I lose network connectivity. The network just drops out, and everything seems fine - no slowdowns, no glitching of apps (which is what happened on my last iMac Pro when the network would drop out). Which is definitely better!

2) I *think* I already had set TSO to 0 last night when the network dropped out. I could be wrong (it may have slipped my mind to do it on that particular boot) but I'm pretty sure I had. In which case maybe there's something else going on

Feb 19, 2018 2:54 PM in response to mustgroove

I am in your exact situations...I wonder if this is happening on other Mac's connected to SMB's through High Sierra. I have experienced the random network drops as well with TSO off. Everything is still unreliable at the moment.

Imac Pro - Crash when copy over 200GB on SMB !?