File Sharing Grinds to a Halt in Yosemite

Question

Level 1

10 points

File Sharing Grinds to a Halt in Yosemite

I've been having a problem since 10.10 whereby out of the blue our Mac Mini fileserver just grinds to a halt and whenever someone tries to access files in the finder on their client Macs it will just hang and beachball.

I've troubleshooted a little bit from having to reboot the server to now ejecting the culprit client off the network This then speeds the network back up again but there is no way to tell which client will be the one that's caused the issue so i have to quickly run around the office and check everyones finder and the problem Mac will be the one that's a tad slower to load files in the finder than others. It's almost as if the computers are DoSing our own server which then slows down the network. There is no way to replicate this issue it seems and comes totally out of the blue.

We mainly have word and excel files on the server. We are connected via SMB with AFP turned off. I don't think this issue happens if we're all on AFP but different issues happen with AFP like not being able to save documents and we also need SMB on.

All clients are on 10.10.3 and the server is also on 10.10.3.

Posted on Apr 30, 2015 2:28 AM

Reply

Answer 1

Best reply

cdhw

Level 4

3,588 points

Apr 30, 2015 4:13 AM in response to Logsi

A corrupt directory on the server can have symptoms like this. I've had good results (on 10.6.8 Server) by rebuilding with Alsop Disk Warrior.

It also reminds me of the problems that occurred back in the day when client-side Spotlight tried to index the server over the network.

Also, if you are running a file server with MS Office documents on it you need to be aware of these issues:

OS X Server & Office Documents - Temporary Lock Files

C.

Reply

Answer 2

Logsi Author

Level 1

10 points

Apr 30, 2015 4:08 AM in response to cdhw

Thanks very much for the information! 🙂

When you say a corrupt directory do you mean OD or just the folder directories?

Thanks for the info about spotlight too, will keep an eye out on that!

I've not been hit with the file locked issue for a very long time, think we encountered that in mountain lion. It's very odd though, every big release fixes one thing and then a new issue arises. Mountain Lion was locked/ghosts files, mavericks was just plain awful and caused massive file sharing problems and yosemite had error 36 and this issue I'm experiencing. Yosemite has been the best thus far though! 😝

Reply

Answer 3

cdhw

Level 4

3,588 points

Apr 30, 2015 5:17 AM in response to Logsi

When you say a corrupt directory do you mean OD or just the folder directories?

The latter, i.e. the directory on the disk that contains the folders that are shared. You could probably rebuild it with some hfs_fsck-fu, but I just give the disk a seeing-to with Disk Warrior, which is now 64bit and can handle very big disks.

Thanks for the info about spotlight too, will keep an eye out on that!

In principle the server can index the shared volumes so client generated traffic shouldn't be a problem. It's all a bit fragile, however, and if something isn't right indexing can really hammer a server.

Can you rule out network issues, such as a failing or overloaded switch? I don't know about SMB, but dropped packets play havoc with AFP performance.

My next move would be to take the gloves off and use tcpdump(1) to capture some SMB traffic.

C.

Reply

Answer 4

Linc Davis

Level 10

209,535 points

Apr 30, 2015 8:11 AM in response to Logsi

Logs?

Reply

Answer 5

Logsi Author

Level 1

10 points

May 6, 2015 6:38 AM in response to Linc Davis

Which logs would be useful? 🙂

Reply

Answer 6

Logsi Author

Level 1

10 points

May 12, 2015 6:59 AM in response to Logsi

May 12 14:35:04 fileserver servermgr_sharing[880]: Metadata.framework [Error]: mdsCopyStoreAttributes failed: (8) (os/kern) no access

May 12 14:35:18 --- last message repeated 3 times ---

The problem happened again and had a quick score of the system log and found this error.

Would this be anything?

Reply

Answer 7

Logsi Author

Level 1

10 points

May 19, 2015 5:03 AM in response to Logsi

Been doing a little more digging because it's totally sporadic when it happens.

Did a tcpdump and opened up in wireshark when things were seemingly slowing down and noticed there is tens, if not hundreds of requests every second via SMB2 protocol from a client computer. The info given for requests usually goes;

Notify Response

Notify Request Error STATUS_INSUFFICIENT RESOURCES

Create Request File

Read Response

Read Request

Literally thousands over the course of the tcpdump which spans only a couple of minutes.

It's not always the same client though. Wondering why it's protocol is SMB2 instead of SMB3.

Reply

Answer 8

cdhw

Level 4

3,588 points

May 19, 2015 5:53 AM in response to Logsi

That SMB status code is saying that there is a shortage of memory on the server:

https://msdn.microsoft.com/en-us/library/ee441884.aspx

Does Application Monitor give any insight here? How much RAM do you have in your server, and how would you characterise the client load?

C.

Reply

Answer 9

Logsi Author

Level 1

10 points

May 19, 2015 5:55 AM in response to cdhw

The server has 16GB ram and never exceeds it. It hovers around 4gb.

Reply

Answer 10

Logsi Author

Level 1

10 points

May 21, 2015 4:17 AM in response to Logsi

Attached a screenshot of a tcpdump. Could this be spotlight indexing? There is hundreds of requests over a few seconds.

Reply

Answer 11

cdhw

Level 4

3,588 points

May 22, 2015 6:07 AM in response to Logsi

If we're guessing, my top two suspects would be the client trying to (a) build a Spotlight index for the share, or (b) include the share in its Time Machine backups.

If we're being systematic, you've made progress, however, because with Wire Shark you can see which client is hammering the server. You may be able to run Activity Monitor on that client and see which process is generating lots of network requests (look at the rate at which packets are increasing for each process), or lots of disk activity. I'm not sure whether SMB2 is encrypted, but I don't think so. If I'm right, you should be able to use Wire Shark to inspect some packets in order to get an idea of which files are being requested by all those SMB requests. E.g. is it always the same one? Are they data files that the user expects to have open? Is the client using an app that puts caches on a server volume, have they decided to put their DropBox folder on your share, etc. ?

C.

Reply

Answer 12

AJ 2010

Level 1

15 points

May 26, 2015 12:03 PM in response to cdhw

Looks like I'm having the same issue. AFP works fine but gradually over time, SMB transfers slow to a crawl. CPU and RAM look great. Rebooting the server fixes the issue for a few hours and then it comes back again.

Reply

Answer 13

AJ 2010

Level 1

15 points

May 26, 2015 12:45 PM in response to AJ 2010

I should also add that there are about 10 AFP clients and about 20 using SMB.

10.10.3

Server 4.1

Reply

Answer 14

Logsi Author

Level 1

10 points

May 27, 2015 7:26 AM in response to cdhw

I've been looking into the whole spotlight index thing some more and disabled it from indexing the network share. Will see if this stops the issue. It's not time machine to my knowledge as some computers that cause the requests aren't backing up.

Looking at a TCPdump has been very valuable though so thank you for that advice cdhw!

Will keep an eye on it. It's such a shame it's so sporadic and I haven't found a way to replicate it my end yet.

The files we use are mainly excel and word files. I think it's usually an excel file that causes the issue.

Reply

Answer 15

May 29, 2015 4:46 PM in response to Logsi

Chiming in here to report that we too are seeing identical symptoms like this on *multiple* servers and separate networks.

Server and clients are both running 10.10.3.

Gradually SMB connections grind way way waaaaaaay down in speed.

CPU and RAM utilization look great - very low.

Reboot of the server "cures" it for several hours to a few days... until it occurs again, and the only thing that fixes it is another reboot.

Reply