1 2 3 Previous Next 30 Replies Latest reply: Aug 8, 2013 7:56 AM by berlindude
berlindude Level 1 Level 1 (0 points)

Hi,

 

I've got a MacPro working as Fileserver (10.6.8 Server) for about 10 Clients (all on OS X 10.6.7 and .8) with Adressbook, AFP, DNS, iCal, OD and SMB running.

The Server is connected over 3 bonded GbE NICs to the Network and has a Promise RAID5 SCSI-320 Storage for the Files.

 

We need to push a lot of big Images (PSD, Tiff, etc.) around over the Network, so this Configuration seemd pretty fast and not too complex. Right now it was working well for over a year (except for minor problems).

 

Two weeks ago I accidentally found a TIFF-Picture with some 108 x 1 px fields on it where the colors were messed up.

 

The only thing i've done with it was to copy it from one AFP-Share to another from my MacPro Client. The original File was still there, so I copied it again and got the same result just on other parts of the Picture (no message, nothing in the Logs). Than I tried to copy a PSD File with some Layers in it (about 800 MB) and the result was some kind of Van Gogh painting (means interesting collors, but the whole picture was messed up).

 

So I started a Testing-Marathon where I checked all the funny stuff like ACLs, POSIX, Cables, Switch, RAM, HDDs, with/without RAID, OSs, OS-Settings, other Clients, direct connetions between the Clients, NIC-Settings, change the com.apple.AppleShare … … … I tested a lot different stuff.

 

Now I can say that the File-Corruption occurs when you use AFP, the two NICs which are communicating send and receive Data at the same time (copy paste between 2 shares) and the Ethernet speed (in activity monitor) is higher than 40 MB/s.

I'm able to reproduce the Corruption with 10.5.7 / 10.6.3, .7, .8 and with Lion.

 

Most tests i've done with a direct connection MBP5,2 - MP5,1 (to reach the speed) but sometimes it worked even with an iMac8,1 - MBP Connection.

No Problem with SMB-Shares, but they are much slower (max. 20 - 30 MB/s).

 

Did I miss something?

Can anybody confirm this behavior?

I'm very grateful for Ideas, I don't have any more and I woldn't like to switch to SMB.


Mac Pro, Mac OS X (10.6.8)
  • 1. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    Two more things i've forgotten:

     

    • I have to copy a min. amount of 1 GB Data to reproduce the behaviour.

     

    • I'm from Germany, so sorry my bad English.

  • 2. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    No ideas?

    Is there no one with a fast gigabit Ethernet connection between two Mac Pros or XServes and two shares on one machine who cold copy paste sth. like a 5 gig DMG or few big PSDs and TIFFs with the other between the shares and check the MD5 checksums?

     

    I found some information, that it could be the checksum offloading to the NIC, but I couldn't find how to deactivate it in OS X.(?)

     

    Right now I'm testing SMB to switch the other Clients, but it's so slow, it hangs sometimes minutes when you're browsing through the folders and worst thing - it doesn't support Spotlight Server.

     

    Any idea is welcome, it could save me from SMB

  • 3. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    Had yesterday a pretty long phone call with an German apple technician. It seems that this is "normal" AFP behavior in combination with newer apple gigabit NICs.

    When communication gets too fast, they could start loosing packets.

     

    The hints to get rid of this are:

    - try playing around with QOS/COS in the switch, maybe it can handle it

    - try a 3rd party NIC

         • which one - they're not allowed to give hints - maybe someone here could help. We need dual GbE / MacPro and OS X compatible (I already know that the sonnet-cards have huge problems when they have to effect performance 24/7)

    - try 10.7 - shure, no problem in a company where everything has to work and people use software which is not 100% compatible to Lion.

         •10.7-Server wouldn't help, because with 10.6 Clients the problem still exists

    - use SMB

  • 4. Re: AFP File corruption when copying from Share A to B
    flyingcactus1 Level 1 Level 1 (0 points)

    I am having a very similar issue - Have you found any other solutions to this issue?

  • 5. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    I already thought no one is reading this and I'm the only one ;-)

     

    Like I already said, the built-in ethernet cards used by apple make this problem when your speed gets too high. So the solution should be (referring to an Apple technician) to decrease the AFP speed with the QOS or COS settings in your switch (if you have one which has this settings).

     

    Right now my problem is: I can't do it, because we've a Netgear L2 switch where the QOS settings are a pain in the a… So every time I changed something, the complete switch crushed and our developers tryed to kill me.

     

    So I stopped testing and started using the much slower SMB (without Spotlight Server!) and wrote an Cisco-switch on the whishlist for my bosses.

     

    I hope this will help, but I can't say right now.

     

    Which switch do you use and what exactly is your issue (you said "very similar")?

  • 6. Re: AFP File corruption when copying from Share A to B
    flyingcactus1 Level 1 Level 1 (0 points)

    We have cisco switches, but not something that I can change the QOS settings on for my departmental server.

     

    The exact issues I am having is when the server has a lot of traffic going to it, (I have not verified an exact value, but it looks like over 40MB/s) there will be file corruption on the files being copied. The servers hosts video and image files for our graphics dept, so every once in a while we will find a video or picture file that looks like parts of the image is missing. 

     

    Since I saw your post, I have tried looking into using ipfw to limit the bandwidth that AFP can use, but no matter what setting I put in, AFP still uses the max bandwidth.

     

    Also I have to have spotlight searching, so switching to SMB is not an option.

     

    Have you tried any 3rd party network cards?  I have one that I might be able to throw in and try out over the next week or two.

  • 7. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    It realy sounds like the same problem.

    The easiest test is a direct connection between two MacPros with an small Cat 5e (or better) cable. Setup two shares on one of them and put some of your larger videos in one of the shares and perform a copy/paste with the other MacPro between the shares on the other. Afterwards do an MD5-check and you should see that they're not identical any more (if copying was fast enough).

     

     

    For ipfw, did you took a look at some tutorials like this one?

    http://info.iet.unipi.it/~luigi/dummynet/

    I didn't tried it yet, but I will.

     

     

    3rd party card:

    We've tried the Sonnet (server) dual GbE card a year ago.

     

    Sorry, I can't say if they also had this problem, because I didn't knew we had it – like you also know, there are no log entries. But I can say that they're not very stable when used 24/7. We broke two of them within thee or four months.

     

    The original Apple dual GbE card has this corruption issue and we've also a stange problem every few weeks. It steps down to 10 Mbit/s (only shown in our switch and in ifconfig, sys-prefs shows 1Gbit and green) and there are no more connections possible until a complete system reboot.

     

    I think both use the (same) Intel-chip? Maybe your card has some other chip. I would appreciate it if you could write your results here.

     

     

    And you're right, working is the **** without spotlight and with SMB. I'm trying it now since three months and I can't allow my users to have to share my pain.

  • 8. Re: AFP File corruption when copying from Share A to B
    pellefri Level 1 Level 1 (0 points)

    Hi Berlindude and Flyingcactus!

     

    I'm in a similar situation as you, having a MacPro server 10.6.8 that is used as mailserver (Kerio) and file server using afp.

     

    Just wanted to hear what has happened and if you have found a solution (better with newer server os...?)

     

    I'm working at a designcompany with about 45 users/coworkers.

    The company has expanded with more people and recently the disk used for the fileserver (about 3 Tb) started getting corrupted (Promise DS4600 as raid5). All disks are fine, the filesystem gets messed up.

     

    Switched to another disk (tried a Synology NAS using iSCSI), same thing happens after a while...

     

    Best regards,

     

    Pelle Fridell

  • 9. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    Hi Pelle,

     

    I wouldn't update the OS X Server higher than 10.6 because with 10.7 Apple decided the Server-System to be a APP :-( !

    So if I understood this right, the new "Server" is just the normal OS X with a little bit more Settings through the Server-APP.

     

    I've tried to copy files with two normal 10.7 Macs and got the same Problem, so I don't think that a newer Server-System will help here (I also don't believe the AFP-Developers are even aware of this Problem).

     

    Right now I'm waiting for our new Cisco SG500 Switch to try getting rid of this by setting bandwidth limits on the Server-LAN-Ports. I'll write here as soon as I get some new Information.

     

    Our long term decision is to change the whole System to Linux. There is neither good Server Hardware nor good Server Software form Apple any more.

  • 10. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    I got the new switch and spend the last three days with searching for the right QoS-Settings for this problem.

    No luck till now.

    The only thing I found is that if you limit the bandwidth to 100 MBit/s the files will be copied perfect. Even with a limit at 150 MBit/s I got MD5 differences after copying.

     

    I've no more Ideas what to try, so I think it's time to surrender.

    Maybe the right answer for this is to use Linux with Netatalk and find a Spotlight-Server tool for it.

  • 11. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    I got a hint to try to compare the files with "diff" so one can see if there is a difference at the same place.

    I tried it and no it's random.

     

    So I made a 1GB null file "head -c 1073741824 /dev/zero >dummy_zero.bin", copied this one around and checked both versions with "cmp --verbose --print-bytes PATH/dummy_zero.bin PATH_TO_COPY/dummy_zero.bin  | tee differences".

     

    Here are my results:

     

    1. Copy – OK

     

    2. Copy:

    1065878937   0 ^@     1 ^A

    1065878938   0 ^@     2 ^B

    1065878939   0 ^@   117 O

    1065878940   0 ^@   343 ?

    1065878946   0 ^@     4 ^D

     

    3. Copy - OK

     

    4. Copy - OK

     

    5. Copy - OK

     

    6. Copy - OK

     

    7. Copy:

    784074137   1 ^A     0 ^@

    784074138   2 ^B     0 ^@

    784074139  66 6      0 ^@

    784074140 164 t      0 ^@

    784074146   4 ^D     0 ^@

     

    8. Copy - OK

     

    9. Copy - OK

     

    10. Copy - Killed my Server (nothing in the Logfiles)

     

     

    I'd say this is a beautyful bug in AFP.

    So if you like your data – never allow your clients to copy files between two shares with GbE-speed!

  • 12. Re: AFP File corruption when copying from Share A to B
    cygnusx Level 1 Level 1 (5 points)

    berlindube, we appear to be experiencing a very similar issue image file corruption (pixels changing in psd files). This happens when copying files from one AFP share to another AFP share, using a client machine, it does not happen if the file copy is made directly using the Server.

     

    What is very worrying is that all copies complete without any error notification from OSX. The dates and file sizes are all correct no indication the copy is not completely the same as the original. I never thought a file could be damaged this way, either it copied or did not, no gray areas of complete copies but the content of the file may have been changed.

     

    We have a small setup, a Mac Mini OSX Server, with Promise Pegasus R4 RAID 6 via Thunderbolt, sharing with 5 client Mac's. Our initial thoughts were Thunderbolt was causing the problems but the direct server copies are fine. So it must be dropped packets via the gigabit ethernet?

     

    The problem appears to get worse if we are using a Mac client which is further away from the Server. The image pixel corruption is more evident than using a Mac client which is closer. Is our ethernet switch not producing enough power to supply the data transfer on these longer lengths of Cat6 cable?

    Its only Netgear desktop switch as its only a small network of machines.

     

    berlindube, you say its an AFP bug so what was your solution?

  • 13. Re: AFP File corruption when copying from Share A to B
    berlindude Level 1 Level 1 (0 points)

    Hi cygnus,

     

    I would say it's not just similar, it's the same issue.

     

    When you copy files directly on the Server, you don't use AFP. So the Data is copied by the Filesystem itself. You're right – no errors, very worrying. An normal network protokoll shoud never allow such behaviour.

     

    Do not try to find wrong pixels in picture-files. Just use some MD5-check software and you will see, the error happens on any file (dmg, tif, psd, zip, …). You only need size and speed while copying.

     

    The distance doesn't matter. I was able to reproduce it by using two fast clients (MacPro) directly attached with an crossover cable (1.5 meter) without an switch. I wold say it's a pure AFP problem.

     

    We're using an workaround:

     

        put all the shares, where clients have to copy data between, in one share together

            when a client copies a file within a share, the data is not tranfered to the client and back. AFP just sends the signal and the copy is done local on the server – so no problem.

            this can be a pain in the a… especially when you have to set komplex ACLs

        if clients need to copy files between shares, they have to copy them on their local HDD first and then on the other share.

     

    I'm working on an usable solution without Apple (Linux Servers, maybe change to Windows Clients).

     

    Actually I'm done with Apple. They only want to sell consumer computers (like Sony) and overpriced telephones. So there is the same amount of concern to get these bugs here fixed like at developing and selling pro-hardware – none!

  • 14. Re: AFP File corruption when copying from Share A to B
    cygnusx Level 1 Level 1 (5 points)

    Hi berlindude,

     

    Thanks for the reply, yesterday I also did exhaustive testing and like yourself reproduced the AFP problem using two Mac's connected directly. I have reported the problem to Apple who have taken reports from our Mac Mini Server, I'm hoping to receive some positive feedback in the next day or so.

     

    If you have issues with a MacPro and we have a Mac Mini then it is not hardware specific. Are you still running OSX Server 10.6.8 as we run OSX 10.8.3 with the Server app, so that must mean the AFP problems have been around for several OSX versions.

     

    I can't understand why more people have not experienced it. Something must trigger this AFP issue because we did not experience the problem with our old Server a PowerMac G5 running OSX Server 10.5 which we replaced with the Mac Mini thunderbolt RAID setup around 6 months ago.

     

    Please can you expand on your workaround, how to achieve this and do the combined shares still appear as separate hard drives on the client Mac's?

     

    put all the shares, where clients have to copy data between, in one share together

            when a client copies a file within a share, the data is not tranfered to the client and back. AFP just sends the signal and the copy is done local on the server – so no problem.

1 2 3 Previous Next