Incoming packets are intermittently dropped by macOS
We are having problems with a Mac Mini that is being used as a build machine for XCode projects. The initial symptoms were that checking out large git projects would randomly stall and then finally fail with a timeout. It also loses connection to our Jenkins machine at random moments.
Digging a little deeper we're seeing intermittent packet loss. We've tried narrowing down the network segment that drops packets and it already happens pinging the IP of the switch the machine is directly connected to. Pinging the Mac from the switch's diagnostics tools shows no packet loss in that direction.
The next step was to do a tcpdump. The 'ping' output will intermittently show 'Request timeout for icmp_seq XXX`, while tcpdump shows that all 'icmp echo request' packets are sent as expected and that the corresponding 'icmp echo response' also arrives very quickly, including those with the sequence numbers that timed out according to the ping command.
Since tcpdump captures the raw packets from the interface, it seems like packets are dropped by the OS somewhere after receiving them.
I would love some suggestions as to how to diagnose this further and find out where the packets are dropped and why. Is there a way to trace the packet through the system and/or get any logging for dropped packets ?
Mac mini (M1, 2020)