Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Mavericks and Failed ARP causing network drops!

I have been wracking my brain about why on our corporate network, after Mavericks upgrade, we start to see dropped packets every 30-60 seconds. Here is an example of that ping.


64 bytes from 10.11.12.13: icmp_seq=135 ttl=63 time=3.705 ms

64 bytes from 10.11.12.13: icmp_seq=136 ttl=63 time=3.473 ms

64 bytes from 10.11.12.13: icmp_seq=137 ttl=63 time=3.811 ms

64 bytes from 10.11.12.13: icmp_seq=138 ttl=63 time=4.110 ms

Request timeout for icmp_seq 139

Request timeout for icmp_seq 140

Request timeout for icmp_seq 141

Request timeout for icmp_seq 142

Request timeout for icmp_seq 143

64 bytes from 10.11.12.13: icmp_seq=144 ttl=63 time=5.417 ms

64 bytes from 10.11.12.13: icmp_seq=145 ttl=63 time=3.587 ms

64 bytes from 10.11.12.13: icmp_seq=146 ttl=63 time=3.744 ms

64 bytes from 10.11.12.13: icmp_seq=147 ttl=63 time=3.486 ms

64 bytes from 10.11.12.13: icmp_seq=148 ttl=63 time=3.466 ms



I think I have found a strange ARPing issue which is causing it. In our corporate environment, we run GLBP (Gateway load balancing protocol) on Cisco gear. As such, the gateway address floats between two devices requiring the mac_addr to change. Looks something like this in the arp table:


efl-ashvartsman:~ ashvartsman$ arp -a

? (10.224.165.1) at 0:7:b4:2:cb:2 on en0 ifscope [ethernet]

efl-ashvartsman:~ ashvartsman$ arp -a

? (10.224.165.1) at 0:7:b4:2:cb:1 on en0 ifscope [ethernet]


On my mountain lion machine, it does a broadcast arp and gets a response for the new mac_addr immediately.


25826.783206000Apple_78:29:ddBroadcastARP42Who has 10.224.165.1? Tell 10.224.165.55
25926.786929000Cisco_e0:ff:40Apple_78:29:ddARP6010.224.165.1 is at 00:07:b4:02:cb:01


This happens seemlessly in the background and no packet loss is observed. However, looks like Mavericks is doing something completely different, and WRONG. It is sending out 5 UNICAST requests back to the mac address it had before (ARP should always be broadcast!!!). It fails these 5 times and then finally does a BROADCAST attempt. Looks like the below. It causes then about a 5 second outage to the network of the machine.


394 67.052366000 Apple_b9:a6:b2 Cisco_02:cb:02 ARP 42 Who has 10.224.165.1? Tell 10.224.165.225

395 68.053450000 Apple_b9:a6:b2 Cisco_02:cb:02 ARP 42 Who has 10.224.165.1? Tell 10.224.165.225

396 69.053595000 Apple_b9:a6:b2 Cisco_02:cb:02 ARP 42 Who has 10.224.165.1? Tell 10.224.165.225

397 70.053893000 Apple_b9:a6:b2 Cisco_02:cb:02 ARP 42 Who has 10.224.165.1? Tell 10.224.165.225

398 71.054363000 Apple_b9:a6:b2 Cisco_02:cb:02 ARP 42 Who has 10.224.165.1? Tell 10.224.165.225

399 72.054466000 Apple_b9:a6:b2 Broadcast ARP 42 Who has 10.224.165.1? Tell 10.224.165.225

400 72.058079000 Cisco_e0:ff:40 Apple_b9:a6:b2 ARP 60 10.224.165.1 is at 00:07:b4:02:cb:01



Here is the arp table during this period:


macsccmtest:~ administrator$ arp -a

? (10.224.165.1) at (incomplete) on en1 ifscope [ethernet]

? (10.224.165.220) at f0:b4:79:21:4c:ec on en1 ifscope [ethernet]



My hunch is that Apple did this to try to reduce bandwidth utilization on the network but it will cause BIG problems on corporate networks that use GLBP or any other protocol to provide redundancy across multiple devices!


Anyone else seeing this? Everyone in my office who has moved to Mavericks can replicate this behavior.

OS X Mavericks (10.9)

Posted on Oct 25, 2013 11:12 AM

Reply
Question marked as Best reply

Posted on Oct 26, 2013 1:45 AM

Same problem here. I spent a lot of time in identifying this problem.


Finally, the solution for this:


$ sudo sysctl -w net.link.ether.inet.arp_unicast_lim=0

net.link.ether.inet.arp_unicast_lim: 5 -> 0


(you can set this also in /etc/sysctl.conf)


This disables the unicast ARP requests.

68 replies

Jan 7, 2014 7:23 PM in response to goft20

Here is the code:

https://raw.github.com/MacMiniVault/Mac-Scripts/master/unicastarp/unicastarp.sh


The first if statement makes surre that the OS is OS X 10.9.x .... not linux or some older version of OS X


If that is true, then the script looks to see if the /etc/sysctl.conf file exists (second if statement).


If that is true, we just look for the word 'unicast' in the file. If we have a positive hit (third if statement) then we stop the script and assume the parameter has already been set and exists.


With that out of the way, the script sets the parameter in the live envrionment. It also uses the 'tee -a' command to take the echo'd line and put it in the config file, appending it if the file exists. We also set the ownership and permissions properly to the config file.

Feb 11, 2014 1:15 PM in response to ashvartsman

This is a bug, but it doesn't appear to be a bug with Mavericks. I believe this is a Cisco bug.


We're having the same problem here with GLBP and Mavericks. The fix provided in this thread solved the problem for us, but I dug a little deeper to determine if this was only limited to Mavericks. From my testing, it does not appear to be.


Linux provides a neat utility called arping that essentially pings a remote host by using unicast arp requests. If you use that tool and ping the lova GLBP address on the router (Not the shared address you use as a default gateway) you'll get consistent ping replies, as you would expect. If you ping the shared address, you get a few (1-3 in my testing) requests answered, and then the remote stops responding. What's interesting is that if you do this a few times, you'll see that the pings stop as soon as the MAC address the router replies with changes. Here's an example :


PROD> $ arping -I em1 192.168.0.1
ARPING 192.168.0.1 from 192.168.0.123 em1
Unicast reply from 192.168.0.1 [00:07:B4:00:01:01] 3.581ms
Unicast reply from 192.168.0.1 [00:07:B4:00:01:02] 3.014ms


The ping stopped as soon as the MAC address changed.


I have an open case with Cisco where I'm referencing this problem as well as this thread. Hopefully more information will follow. If I receive any viable information, I'll pass it on.

Feb 12, 2014 1:19 PM in response to ashvartsman

So I spoke with Cisco and they're pointing a finger at Apple. Per Cisco, Apple has implemented this improperly. They claim that the MAC Address is being removed from the ARP table prior to the Unicast ARP requests. Can anyone confirm this is the case?


They also pointed out this page which also explains the issue and has a note that 10.9.2 is rumored to resolve this. http://www.macstadium.com/blog/osx-10-9-mavericks-bugs/

Feb 12, 2014 7:16 PM in response to XenoPhage

I have a friend who has a new MacBook Pro. When it was running 10.9.0, everything was fine. But when we updated it to 10.9.1, the wireless would not hold a connection for more than a minute or so even though he was sitting right next to his time capsule. He spent almost two hours with a very responsive guy at the local Genius Bar. The Genius finally decided to roll the machine back to 10.9.0 and wait for the next release of 10 to see if the problem will be fixed. Once the machine was installed again with 10.9.1, everything is fine. I don't know if this has anything to do with your issue or not. BTW, I run 10.9.1 on my older MBP with no issues. Phil

Feb 28, 2014 6:44 AM in response to jonschwenn

I recently changed my router and now it connects and stays connected but I have no internet access intermittently or until I turn wifi off and back on. This new router is a Samsung SMT-G7400/XEN and I think it's dualband.


How would I be able to fix this issue? I'm not even sure it's what other people here are having because most of what everyone is talking about goes over my head. I'm on a late 2013 13" MBPr on 10.9.2.

Mavericks and Failed ARP causing network drops!

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.