AFP File corruption when copying from Share A to B

Two more things i've forgotten:

• I have to copy a min. amount of 1 GB Data to reproduce the behaviour.

• I'm from Germany, so sorry my bad English.

No ideas?

Is there no one with a fast gigabit Ethernet connection between two Mac Pros or XServes and two shares on one machine who cold copy paste sth. like a 5 gig DMG or few big PSDs and TIFFs with the other between the shares and check the MD5 checksums?

I found some information, that it could be the checksum offloading to the NIC, but I couldn't find how to deactivate it in OS X.(?)

Right now I'm testing SMB to switch the other Clients, but it's so slow, it hangs sometimes minutes when you're browsing through the folders and worst thing - it doesn't support Spotlight Server.

Any idea is welcome, it could save me from SMB 😉

Had yesterday a pretty long phone call with an German apple technician. It seems that this is "normal" AFP behavior in combination with newer apple gigabit NICs.

When communication gets too fast, they could start loosing packets.

The hints to get rid of this are:

- try playing around with QOS/COS in the switch, maybe it can handle it

- try a 3rd party NIC

• which one - they're not allowed to give hints - maybe someone here could help. We need dual GbE / MacPro and OS X compatible (I already know that the sonnet-cards have huge problems when they have to effect performance 24/7)

- try 10.7 - shure, no problem in a company where everything has to work and people use software which is not 100% compatible to Lion.

•10.7-Server wouldn't help, because with 10.6 Clients the problem still exists

- use SMB 😁

I already thought no one is reading this and I'm the only one ;-)

Like I already said, the built-in ethernet cards used by apple make this problem when your speed gets too high. So the solution should be (referring to an Apple technician) to decrease the AFP speed with the QOS or COS settings in your switch (if you have one which has this settings).

Right now my problem is: I can't do it, because we've a Netgear L2 switch where the QOS settings are a pain in the a… So every time I changed something, the complete switch crushed and our developers tryed to kill me.

So I stopped testing and started using the much slower SMB (without Spotlight Server!) and wrote an Cisco-switch on the whishlist for my bosses.

I hope this will help, but I can't say right now.

Which switch do you use and what exactly is your issue (you said "very similar")?

It realy sounds like the same problem.

The easiest test is a direct connection between two MacPros with an small Cat 5e (or better) cable. Setup two shares on one of them and put some of your larger videos in one of the shares and perform a copy/paste with the other MacPro between the shares on the other. Afterwards do an MD5-check and you should see that they're not identical any more (if copying was fast enough).

For ipfw, did you took a look at some tutorials like this one?

http://info.iet.unipi.it/~luigi/dummynet/

I didn't tried it yet, but I will.

3rd party card:

We've tried the Sonnet (server) dual GbE card a year ago.

Sorry, I can't say if they also had this problem, because I didn't knew we had it – like you also know, there are no log entries. But I can say that they're not very stable when used 24/7. We broke two of them within thee or four months.

The original Apple dual GbE card has this corruption issue and we've also a stange problem every few weeks. It steps down to 10 Mbit/s (only shown in our switch and in ifconfig, sys-prefs shows 1Gbit and green) and there are no more connections possible until a complete system reboot.

I think both use the (same) Intel-chip? Maybe your card has some other chip. I would appreciate it if you could write your results here.

And you're right, working is the **** without spotlight and with SMB. I'm trying it now since three months and I can't allow my users to have to share my pain.

Hi Pelle,

I wouldn't update the OS X Server higher than 10.6 because with 10.7 Apple decided the Server-System to be a APP :-( !

So if I understood this right, the new "Server" is just the normal OS X with a little bit more Settings through the Server-APP.

I've tried to copy files with two normal 10.7 Macs and got the same Problem, so I don't think that a newer Server-System will help here (I also don't believe the AFP-Developers are even aware of this Problem).

Right now I'm waiting for our new Cisco SG500 Switch to try getting rid of this by setting bandwidth limits on the Server-LAN-Ports. I'll write here as soon as I get some new Information.

Our long term decision is to change the whole System to Linux. There is neither good Server Hardware nor good Server Software form Apple any more.

I got the new switch and spend the last three days with searching for the right QoS-Settings for this problem.

No luck till now.

The only thing I found is that if you limit the bandwidth to 100 MBit/s the files will be copied perfect. Even with a limit at 150 MBit/s I got MD5 differences after copying.

I've no more Ideas what to try, so I think it's time to surrender.

Maybe the right answer for this is to use Linux with Netatalk and find a Spotlight-Server tool for it.

I got a hint to try to compare the files with "diff" so one can see if there is a difference at the same place.

I tried it and no it's random.

So I made a 1GB null file "head -c 1073741824 /dev/zero >dummy_zero.bin", copied this one around and checked both versions with "cmp --verbose --print-bytes PATH/dummy_zero.bin PATH_TO_COPY/dummy_zero.bin | tee differences".

Here are my results:

1. Copy – OK

2. Copy:

1065878937 0 ^@ 1 ^A

1065878938 0 ^@ 2 ^B

1065878939 0 ^@ 117 O

1065878940 0 ^@ 343 ?

1065878946 0 ^@ 4 ^D

3. Copy - OK

4. Copy - OK

5. Copy - OK

6. Copy - OK

7. Copy:

784074137 1 ^A 0 ^@

784074138 2 ^B 0 ^@

784074139 66 6 0 ^@

784074140 164 t 0 ^@

784074146 4 ^D 0 ^@

8. Copy - OK

9. Copy - OK

10. Copy - Killed my Server (nothing in the Logfiles)

I'd say this is a beautyful bug in AFP.

So if you like your data – never allow your clients to copy files between two shares with GbE-speed!

Hi cygnus,

I would say it's not just similar, it's the same issue.

When you copy files directly on the Server, you don't use AFP. So the Data is copied by the Filesystem itself. You're right – no errors, very worrying. An normal network protokoll shoud never allow such behaviour.

Do not try to find wrong pixels in picture-files. Just use some MD5-check software and you will see, the error happens on any file (dmg, tif, psd, zip, …). You only need size and speed while copying.

The distance doesn't matter. I was able to reproduce it by using two fast clients (MacPro) directly attached with an crossover cable (1.5 meter) without an switch. I wold say it's a pure AFP problem.

We're using an workaround:

put all the shares, where clients have to copy data between, in one share together

when a client copies a file within a share, the data is not tranfered to the client and back. AFP just sends the signal and the copy is done local on the server – so no problem.

this can be a pain in the a… especially when you have to set komplex ACLs

if clients need to copy files between shares, they have to copy them on their local HDD first and then on the other share.

I'm working on an usable solution without Apple (Linux Servers, maybe change to Windows Clients).

Actually I'm done with Apple. They only want to sell consumer computers (like Sony) and overpriced telephones. So there is the same amount of concern to get these bugs here fixed like at developing and selling pro-hardware – none!

I have reported the problem to Apple who have taken reports from our Mac Mini Server, I'm hoping to receive some positive feedback in the next day or so.

I have also reported it (one or two years ago), don't expect any help …

Are you still running OSX Server 10.6.8 … the AFP problems have been around for several OSX versions.

Yup, still 10.6.8 – don't believe in Apps ;-)

I startet reproducing it with 10.5.7 (client-version) and it's not hardware specific (had this also with an iMac and a MBP).

I can't understand why more people have not experienced it.

They do, but they don't see it.

Something must trigger this AFP issue because we did not experience the problem with our old Server …

Like I already said, It's the speed. You need a min. of 40 MB/s in both directions at the same time. Our second server is a G5 PowerMac and I never got this speed out of it.

Please can you expand on your workaround

E.g. you have the shares A, B and C (folders a, b and c on the RAID), take the shares away, make a new folder d and the share D and put the folders a, b and c in d.

and do the combined shares still appear as separate hard drives

This you can put to "pain in the a…" – you can't any more. The clients see separate folders, maybe with separate restrictions but no more separate shares 😟