Incoming packets are intermittently dropped by macOS

We are having problems with a Mac Mini that is being used as a build machine for XCode projects. The initial symptoms were that checking out large git projects would randomly stall and then finally fail with a timeout. It also loses connection to our Jenkins machine at random moments.


Digging a little deeper we're seeing intermittent packet loss. We've tried narrowing down the network segment that drops packets and it already happens pinging the IP of the switch the machine is directly connected to. Pinging the Mac from the switch's diagnostics tools shows no packet loss in that direction.


The next step was to do a tcpdump. The 'ping' output will intermittently show 'Request timeout for icmp_seq XXX`, while tcpdump shows that all 'icmp echo request' packets are sent as expected and that the corresponding 'icmp echo response' also arrives very quickly, including those with the sequence numbers that timed out according to the ping command.


Since tcpdump captures the raw packets from the interface, it seems like packets are dropped by the OS somewhere after receiving them.


I would love some suggestions as to how to diagnose this further and find out where the packets are dropped and why. Is there a way to trace the packet through the system and/or get any logging for dropped packets ?

Mac mini (M1, 2020)

Posted on May 9, 2023 9:25 AM

Reply

Similar questions

10 replies

May 9, 2023 11:07 AM in response to etresoft

Again, if the switch is the problem why do I see perfectly valid responses ? There is no actual packet loss on the network, all pings are answered as confirmed by a capture from the ethernet interface. I tested several difference clients to confirm this.


I’ve done some additional testing:


The problem *only* occurs when packets are large enough to get fragmented. And only intermittently. If I ping with a payload that causes the packets to fragment, some packets are dropped. If I ping with a smaller payload, no problem.


A tcpdump and analysis with Wireshark shows all packets are sent out correctly and responded to correctly. No fragments are missing and all packets can be reconstructed without error by Wireshark, including the ones that ‘ping’ fails to receive.

May 9, 2023 9:59 AM in response to Aaargh79

Aaargh79 wrote:

If you had read the whole thing, you’d see that the problem is not the switch. The responses actually *do* arrive without delay. This is a software problem, not a network hardware problem.

I read the whole thing. It sounds like a case of premature diagnostics. You have to first isolate the problem. You have at least 5 different devices with up to 5 different operating systems. The most likely candidate is the switch. Since the switch is a core component that is difficult to swap out, you should try changing the client first.

Feb 7, 2024 1:33 PM in response to Aaargh79

I found this thread searching for a similar issue with my Mac Studio 10G Ethernet (Speed: 5000Base-T, Duplex: full-duplex, MTU: Standard (1,500), AVB/EAV mode: Disabled).


When my Mac returns from sleep sometimes the Ethernet connection under System Preferences shows up as being connected while all TCP/IP traffic seems to have stopped. I can't access my switch or router via any browser with their IP. Practically it looks like the ethernet connection is down.


But I can still send pings. When I ping my switch and router both of them show sporadic ping timeouts.


Can you tell me what tcpdump commands you used to determine that there's no actual packet loss? Maybe we can compare notes this way.


I already filed several tickets in the Apple Feedback Assistant.


When I deactivate and reactivate the Ethernet connection via System Preferences everything is back to normal for a while.


Funny thing is, I have a second Mac Studio same configuration with 10G Ethernet connected to the same switch which never shows this behaviour. I already switched ports between both of them. It didn't change anything. The issue is isolated to my Mac Studio.


For reference here's the output I see when I ping my switch and router when the issue occurs. They both have a similar ping pattern.


64 bytes from 192.168.10.1: icmp_seq=0 ttl=64 time=1.102 ms
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
64 bytes from 192.168.10.1: icmp_seq=6 ttl=64 time=0.943 ms
Request timeout for icmp_seq 7
Request timeout for icmp_seq 8
64 bytes from 192.168.10.1: icmp_seq=9 ttl=64 time=1.023 ms
Request timeout for icmp_seq 10
Request timeout for icmp_seq 11
Request timeout for icmp_seq 12
Request timeout for icmp_seq 13
Request timeout for icmp_seq 14
Request timeout for icmp_seq 15
Request timeout for icmp_seq 16
64 bytes from 192.168.10.1: icmp_seq=17 ttl=64 time=0.993 ms
Request timeout for icmp_seq 18
Request timeout for icmp_seq 19
64 bytes from 192.168.10.1: icmp_seq=20 ttl=64 time=0.801 ms
64 bytes from 192.168.10.1: icmp_seq=21 ttl=64 time=0.990 ms
Request timeout for icmp_seq 22
64 bytes from 192.168.10.1: icmp_seq=23 ttl=64 time=0.894 ms
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 192.168.10.1: icmp_seq=26 ttl=64 time=0.869 ms
64 bytes from 192.168.10.1: icmp_seq=27 ttl=64 time=0.863 ms
Request timeout for icmp_seq 28
Request timeout for icmp_seq 29
Request timeout for icmp_seq 30
64 bytes from 192.168.10.1: icmp_seq=31 ttl=64 time=1.383 ms
64 bytes from 192.168.10.1: icmp_seq=32 ttl=64 time=0.890 ms
64 bytes from 192.168.10.1: icmp_seq=33 ttl=64 time=1.512 ms
64 bytes from 192.168.10.1: icmp_seq=34 ttl=64 time=0.990 ms


May 9, 2023 12:06 PM in response to Aaargh79

I didn't say the switch was the problem. I said it was most likely the problem since all devices are connected to it and you see no problem between the switch and the mini.


If those responses were "perfectly valid", then you wouldn't have been posting this question in the first place. Something in your network isn't happy. By default, everything should work perfectly. Macs are relatively difficult to configure in the first place. So by extrapolation, they are relatively difficult to configure incorrectly. The exact opposite is true for the switch. Furthermore, the switch is relatively easy to swap out, change ports, change cables, bypass entirely, etc.


But if Wireshark says there is no problem, then that proves that you have no problem, right?

May 9, 2023 12:44 PM in response to etresoft

> If those responses were "perfectly valid", then you wouldn't have been posting this question in the first place


I'm posting in a 'macOS' discussion group because I suspect the problem is something in macOS. I would be posting in a networking group if I suspected it was a network problem.


> Something in your network isn't happy.


Again, the problem is not the network itself. The correct packets are sent out, and the correct responses arrive back. The responses go from A (the switch) to B (the NIC) to C (the app). No need to look for something wrong on the path from A to B when they safely arrive at B and go missing somewhere on the path from B to C.


It could be something in the network stack on macOS, and that's what I'm trying to figure out. I'm not familiar enough with macOS networking to diagnose this, on Linux it would be relatively simple to trace the packets through ipfilter and see where the end up. I'm looking for help on how to do that on macOS.


The simple facts are: the packets arrive at the network interface, a tcpdump proves this. The packets don't arrive at the application, so they are dropped somewhere along that path, for some reason. I'm trying to figure out where they are dropped and why.


> Macs are relatively difficult to configure in the first place. So by extrapolation, they are relatively difficult to configure incorrectly. The exact opposite is true for the switch.


The mac is actually a relatively complicated setup. It runs a VPN client and several VM's. By contrast, the switch is a simple layer 2 switch, no fancy config at all. It's a brand new out of the box Aruba 1830 with the default config which makes it behave like a basic dumb switch.

May 9, 2023 1:35 PM in response to Aaargh79

Aaargh79 wrote:

The mac is actually a relatively complicated setup. It runs a VPN client and several VM's.

Call me crazy, but maybe you ought to have led with that. 😄


I don't know what to tell you. If you are running a consumer VPN, then nobody's going to give you the time of day. It was never designed for that. If you are running an enterprise VPN, then, in theory, you have some support. I would be shocked if your enterprise VPN support technicians knew anything about the Mac or cared in the slightest. Most likely, they will just blame Apple and be done with it.


And then there's the VMs. Are you talking about problems connecting to one of the VMs? Multiple VMs? The host? What VM product are you using? How is the networking on those VMs configured? A year ago, you might have had a bit more interest from your VM vendor support. But recent developments have wrecked the Mac VM market. You're on your own there, too.


You realize this is a user-to-user tech support forum for consumer Apple products? It sounds like you are hoping for some known problem with macOS dropping packets? Sorry. News to me.

May 9, 2023 2:15 PM in response to etresoft

> I don't know what to tell you. If you are running a consumer VPN, then nobody's going to give you the time of day.


It's not. The VPN is to securely connect to our cloud environment, which is where our git repository is hosted, where our CI automation lives, where build artefacts are stored, etc.


> I would be shocked if your enterprise VPN support technicians knew anything about the Mac or cared in the slightest.


The VPN is only used for a specific subnet, none of this test traffic is routed through the VPN and shouldn't even touch it. I even shut down the VPN to make sure if can't interfere.


> Are you talking about problems connecting to one of the VMs


No. This is a problem on the host. It only indirectly affects the VM's. The VM's aren't even running as they are launched on demand.


The only reason I mention the VPN and VM's is that both add a few virtual interfaces. Although neither should affect any of these tests as this traffic is routed directly to the ethernet interface they do make some modifications to the network configuration. However, I can't think of a way they could affect this.


> You realize this is a user-to-user tech support forum for consumer Apple products?


I was hoping these forums were frequented by more advanced 'users' who have dealt with similar issues. Unfortunately Apple requires apps to be built using Xcode on macOS, which means running our CI builds on consumer-grade hard and software. I'd much prefer to run all this on a Linux instance in the cloud, but unfortunately we can't.

May 9, 2023 7:21 PM in response to Aaargh79

Ok. Maybe back up a bit.


What do I know? You have a Mac mini used as an Xcode build and VM host. That pretty much eliminates the PPC models, so this machine dates from no earlier than 2006, most likely at least 2009, to 2023. I would have assumed it to be an Intel machine, but when Apple Silicon was first introduced, it seemed like people were only using them for VMs. Your post sig says M1 2020, but those signatures aren’t reliable.


This mini is connected via Ethernet to a switch. It is connected to some (remote?) git, Jenkins, and possibly file servers of some kind. At least some of these connections are using a VPN that you assure us is not consumer-grade and irrelevant to any networking problems.


This mini is experiencing random loss of networking episodes of unknown duration. This problem began at some point between 2006 and May 9th, 2023.


You can detect no problems when running diagnostics from the switch. When running diagnostics from some other computer of unknown vendor and configuration, possibly on some other network, you are detecting random loss of some large packets, but only from the OS perspective. Your diagnostic tools Wireshark and tcpdump report no problems.


What don’t I know? Specific model of mini. Configuration and performance characteristics. Is it crashing or running out of RAM during these episodes? The time that these problems started happening. Any configuration changes to the mini that occurred immediately before the problem. The current OS version of this computer. The specific VPN vendor and architecture. The specific VM vendor and guest systems. Whether these diagnostics and packet losses were detected on the mini itself or some other computer. The vendor, configuration, OS version of said diagnostic client. The network connection between this computer and the mini. You’ve said it isn’t over the VPN. But is it also directly connected to the same switch? Are these half-loss of packets being detected actually the casus belli of the network disconnects experienced on the mini?


I’m sorry, but I just don’t have enough information to make any kind of guess. I have heard of problems with people trying to do split tunnels on certain commercial VPNs where they will lose all connectivity. That doesn’t sound like the same problem, but it could be related. Modern Mac networking is not straightforward. People assume it is like Linux or BSD, and it isn’t. You have more information, tools, and access than anyone here. You sound more competent than most IT folks. I’m not confident that your diagnostic tools are all that reliable. Wireshark seemed ignorant of modern Mac configurations for several years. I don’t know if that has ever changed.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Incoming packets are intermittently dropped by macOS

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.