Imac Pro - Crash when copy over 200GB on SMB !?

Question

ODY1980 Author

Level 1

9 points

Imac Pro - Crash when copy over 200GB on SMB !?

Hello, I have a really strange problem - since I upgrade to my new imac pro.

I really hoped for the 10.13.3 update - because it mentioned to fix SMB problems with crashes - but sadly it didn't help.

The problem I have is the following:

When I copy my VM Ware Folder (via 1Gbps ethernet cable) as a backup to a synology via SMB as soon as I hit around 200GB over the network the network of the imac stalls. No more receiving/sending - no more possibility to ping - even trying to disable and reenable the network doesn't work. No errors shown in any network utlity of bad packets etc. If I just wait then the mac just reboots itself without any error message and comes back.

What I tried so far:

1) Another imac Pro - same issue!
2) Even though the lockup happens on the imac (so it can't be solely related to the network itself) i tried to check if there is anything wrong in the network. Changed router, Change switch, Changed network cable etc. ! However with an old imac I can copy the file on the same line easily without any issues.

3) Tried different SMB settings 2.0 / 3.0 / disabled signing etc. - everything without any help.

For me its clear there is a severe driver issue with the new 10GB network port on the imac pros - but how can I truly identify the issue and report it to apple - so they take me serious without doing the normal troubleshooting...

If anyone has an idea, I would be more than happy to hear it 🙂

Like i said if i for example copy 50GB and then later 150GB - it will happen at 150GB - so it look like something is running full in a some kind of buffer at 200GB copy over network to smb.

Kind regards,
Oliver

macOS High Sierra (10.13.3), null

Posted on Jan 24, 2018 12:29 AM

Reply

Answer 1

Top-ranking reply

ODY1980 Author

Level 1

9 points

Jan 24, 2018 8:44 AM in response to ODY1980

Hey there - with your help I was finally able to find the issue.

Manually did not help me in the beginning because 1000T was correctly set by automatic. But automatic also sets the full duplex mode to:

full-duplex, flow-control, energy-efficient-ethernet

The problem here is the last option -> energy-efficient-ethernet <- this causes when using the SMB protocol and copying more than 200GB of data to the network to completly crash or stall the imac pro.

So all you need to do is changing it to manual and setting the full duplex to: full-duplex, flow-control

without this energy-efficient-ethernet - this also boosts your network speed by 2-3mb/sec.

I reported this to apple and hopefully they will make a fix with the automatic setting.

Kind regards,

Oliver

Reply

Answer 2

ODY1980 Author

Level 1

9 points

Feb 3, 2018 2:26 PM in response to ODY1980

After a lot more investigation of this problem and due to the fact that it reappeared for me since I reproduced it for apple - it seems like I finally found the real culprit of this issue. After reinstalling the mac the problem was still persistent - I couldn't even do a time machine restore with the imac pro via my SMB backup due to this critical network issue - so I can imagine there are a lot of people out there and pulling their hairs off why their machines are crashing or why their time machines backups won't get restored over a SMB network...

Interestingly the energy-efficient-ethernet seems not to be the issue - it must be somehow related but the real issue is the new TCO (TCP segmentation offload) - a feature which offloads work from the CPU to the network card in order to safe processor power. However this feature was already be known at some point to cause issues on large file transfers.

I found this by coincidence when comparing why the external apple gigabit ethernet adapter didn't cause any issues at all but the internal in the imac pro does. So this ethernet option was the only difference! It may also explain why people with the sonnet 10gb adapter experience the same - pretty sure that this card also uses TCO !

So in order to disable that all you have to do is typing the following command into the terminal:

sudo sysctl -w net.inet.tcp.tso=0

This will immediately turn TCO off - no need to restart the network etc. ! However if you are doing a reboot you need to enter this command again. I don't know on how to make this automatic yet - since it requires sudo rights and can't be simply executed as a login script. But I think this will help apple track this down pretty easily and hopefully they will apply a fix for that soon enough.

Please try it out and report back here if it worked for you or not!

Reply

Answer 3

spertti

Level 1

36 points

Jan 24, 2018 2:57 AM in response to ODY1980

One thing to try, as I had samekind problem.

Set your network card manually 1gb, not automatically.

System settings-> Network-> ethernet -> advanced -> hardware- >configure -> MANUALLY

Reply

Answer 4

ODY1980 Author

Level 1

9 points

Jan 30, 2018 7:28 AM in response to Wunderlicht

Yep they definately have issues here.

But on my end it was exactly at the 200GB mark - after a fresh restart - where the freeze happens. So after a restart copying 200GB to SMB would crash at almost exactly this amount or no crash but the network connection stalled.

However since disabling the energy-efficient duplex mode all is fine. So that's really weird - that you could not fix it with this. An external network card also solves the issue (I tried it and it worked). But hopefully they will fix it with next update.

Reply

Answer 5

Feb 1, 2018 2:04 PM in response to IndianaPwns

Got a really nice screen session and overseas talk with Mike F. from Apple. In over 30 minutes we tried to identify the error and crash, and state of the hardware here.

Since further investigations on myself the error (on my iMac pro) will happen with AFP, CIFS or SMB connections (against saing previously cifs and AFP were ok) after or around copying 200 GB.

Until the ~200GB were reached. You can split the copy job e.g. 100GB in the morning and 100 at the afternoon (without rebooting), the error still happened after around 200 GB were reached in total.

Network settings: Can't prevent freeze/crash/reboot with network-energy settings enabled or disabled, other flow control, different frame sizes (1500 or 9000) or leaving settings 'automatic' or by any other manual setting. This seems slightly being different from some machines, reading this so far here.

But the error/crash will NOT happen, when you use a FTP program. Still done my copying (changing NAS harddisks) with this kind of program these days, to get it finished... (700 GB so far in one session without any crash)

Current state: Will having a second screen session with Mike F. from Apple, Sacramento on monday and transferring a system wide full machine log to apple, after reproducing a 'fresh' crash for further examination by Apple.

Good to hear that something is happening inside the apple space station, and were were heared 🙂

If they can reproduce it, proably at the next update(s) the error will be fixed.

Reply

Answer 6

Feb 4, 2018 6:22 AM in response to ODY1980

I think this is it!!

My testing:

- changed tso-parameter to: 0

- mapping a SMB connection (to local NAS)

- copied around 400 GB without any stuck/disconnect or crash

- disconnected the smb share

- switch the tso-parameter back to default (1)
(no reboot between any step)

- reconnected by smb to the same machine and folder (NAS)

- started a new copy job (with the same files)
After around early 50GB the network connection was kicked (disconnected) and the machine is getting more unresponsive.
Done then a manual reboot to prevent corrupted files, before the machine get a kernel panic.

On my opinion this, the net.inet.tcp.tso parameter (or something at the network driver in bond with the tso 'thing') has to be the imac pro fault with network shares.
Good job ody1980 👍

Possible any other can confirm this?
At monday afternoon there's a planned call with Mike from the Apple Engeneering Department (In the meantime, had sended them a machine wide log after crash). If he's calling through again, i'll get him a hint on that.

Reply

Answer 7

Feb 1, 2018 12:43 PM in response to ODY1980

I'm not so sure...when I set the iMac Pro's Ethernet settings to "Automatically" it fills the Duplex options with energy-efficient Ethernet. That's when I'm experiencing crashing...

We DID transfer 5TB last night on a separate 2016 iMac with no issues - so I'm going to say it's NOT our SMB/Server's issue.

Next step: We're going to set the configuration to Manual tonight with 1000baseT with NO energy-efficient ethernet and see if it gets over 200GB before crashing.

Will get back to this thread! Apple - you there?

Reply

Answer 8

Feb 6, 2018 2:35 PM in response to ODY1980

So, it seems that this is the golden bullet for this particular problem!

Yesterday, my replacement iMac Pro arrived (I was asked by Apple to return the first one even though we thought there was a solution at that point 'just in case') and I took it to my studio last night, entering the command to turn of TCP.TSO

This morning, I got a call from Apple, and we went through the steps required with the replacement iMac Pro to reproduce the issue... and bam... as soon as TCP.TSO was turned on, back came the problem!

Since ending the call, I've turned off TCP.TSO again, and the machine has worked flawlessly. So here's hoping the Apple Engineers make a patch ASAP to fix this annoying issue!

Nikki.

P.S. Now the thing isn't crashing, I'm really enjoying how quickly this 10-core 128GB machine slices through FCPX footage like butter. Oh yeah. Mmm. Butter...

Reply

Answer 9

ODY1980 Author

Level 1

9 points

Feb 18, 2018 11:26 PM in response to mustgroove

Yep that's why a new iMac Pro won't help you 🙂

The issue is solely related to the network adapter and this TSO. But if you set it to 0, you should not get those reboots and your network should not start to stop functioning.

Kind regards,
Oliver

Reply

Answer 10

Feb 19, 2018 5:17 PM in response to mustgroove

Just had a fantastic conversation with an Apple Enterprise Support Agent. It looks like High Sierra isn't playing nice (at all) with SMB. They have received a lot of calls and are aware of all the issues. They are working hard on a fix. The agent told me the newest 10.13.4 High Sierra update should fix 99% of them! HOOORAY!!!

Reply

Answer 11

Feb 5, 2018 1:28 PM in response to Wunderlicht

Yup! I'm finding the same kind of results! So it seems that net.inet.tcp.tso is the issue!

Honestly @Ody1980, I could hug you right now. I mean, this seems to have fixed my issues completely!

My 'replacement' iMac Pro is currently sitting at the local Apple Store. I wonder if I'll even bother going to pick it up now, because honestly, I don't want to go through the hassle of swapping out a machine that doesn't NEED replacing. I need to get Apple to OK this though, because they wanted to see my machine.

As for offloading processing to the network card? Well, I really don't think this 10-core is going to suffer much of a performance drop...

Reply

Answer 12

ODY1980 Author

Level 1

9 points

Jan 31, 2018 11:54 AM in response to IndianaPwns

Try using apples external Gigabit Ethernet Adapter - if the problem is still happening, then you know it is not related to this specific problem.

Reply

Answer 13

Jan 30, 2018 7:23 AM in response to Wunderlicht

Done with the tests so far.

Conclusion:
- Connection via AFP to NAS: 250GB++ all ok.
- Connection via CIFS to NAS: 400GB++ all ok.

- Switching to AFP protocol to NAS: Nearly system freeze after around 100 GB copy NAS-to-NAS (or copying iMac to NAS vice versa. Switching curently hard disks).

While doing the tests no reboot were proceeded.

So definitely the SMB connection(s) with a lots of copying (100-200GB or more) are having issues with the iMac pro.
Apple, do you ready this? It's reproducable. 😐

Neither any different network setting are helping (at least, switching from "automatic" to manual and only fullduplex (with no additional settings) ... also trying jumbo frames. The fault must be at the SMB connection/protocol, not (directly) at network settings.

Reply

Answer 14

ODY1980 Author

Level 1

9 points

Feb 1, 2018 2:30 PM in response to Wunderlicht

Really strange that even your issue looks almost the same as mine looked - except that I could not reproduce it on AFP or other protocols and also the energy-efficient-ethernet option resolved it.

But overal your issue looks very similar - I had even had the same problem with the 200GB it didn't matter if you did it in one go or seperated - but once after a clean restart when you reached over 200GB SMB transfer the imac pro stalled.

I am also at it with an engineer from apple, so let's hope they can resolve this soon.

Reply

Answer 15

ODY1980 Author

Level 1

9 points

Feb 6, 2018 5:45 AM in response to IndianaPwns

Well I don't have a 10GB network, but for a normal gigabit ethernet - there shouldn't be any reduced network speed.

I am even getting 2-3MB/sec more with this setting from 113Mb/sec to 116Mb/sec - which is the max you can get over a 1Gbps network.

You sure that this is related to this option? You can easily change this setting on the fly in the terminal from 0 back to 1 and you should immediately see the difference in network speed. There is no need to reconnect to the SMB nor restarting the network. TSO is applied/disabled instant. Thus you could easily track down if its really this option what is causing this.

Kind regards,

Oliver

Reply