Looks very much like the iPhone is having a very hard time receiving packets. I managed to capture the some logs on the AP where the iPhone stubbornly does not respond to any ARP requests.
Note that 10.101.2.100 is the iphone in question. Eventually, the device responds after 30 seconds, and the page loads successfully.
Ethernet MAC addresses blanked out to (00:00:00) on purpose...
[root@odo ~]# tcpdump -i WIFI arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on WIFI, link-type EN10MB (Ethernet), capture size 65535 bytes
20:26:09.076547 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:10.076576 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:11.076572 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:15.657550 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:16.657595 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:17.657613 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:18.666570 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:19.666639 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:20.666612 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:22.671571 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:23.588396 ARP, Request who-has 10.101.2.1 tell 10.101.2.42, length 46
20:26:23.588425 ARP, Reply 10.101.2.1 is-at 00:0a:cd:00:00:00 (oui Unknown), length 28
20:26:23.671570 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:24.671568 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:26.909550 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:27.909567 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:28.772621 ARP, Request who-has 10.101.2.1 tell 10.101.2.100, length 46
20:26:28.772648 ARP, Reply 10.101.2.1 is-at 00:0a:cd:00:00:00 (oui Unknown), length 28
20:26:33.772568 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:34.772578 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:35.772567 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:39.279586 ARP, Request who-has 10.101.2.100 tell 10.101.2.1, length 28
20:26:39.333825 ARP, Reply 10.101.2.100 is-at 80:be:05:00:00:00 (oui Unknown), length 46
To verify the theory that this is MAC address related, I tried installing a static entry in the router. This solved the ARP request timeout by eliminating the need for ARP entirely, but the connectivity is still shoddy at best.
Additionally, I was able to capture very similar behavior on in-bound TCP data while trying watch a youtube video (which, by the way, would never load. resulted in error messages):
Here is the wireshark analysis. Pay attention to those TCP sequence numbers. We can see data payload with TCP sequence numbers 183334, 184778, 186222, and 187666 get transmitted from AP to iPhone in frames 2381, 2382, 2383, and 2384, respectively. Rather than sending out ACK for these packets, we see the iphone send out SACK ending at sequence numbers 183334. We can see these SACKs in frames 2431 2433 2434 2436, and yet again in 2453 2454 2455 2456 and again in 2462 2463 2466 2467. Finally we see data payload re-transmission from Google server in frames 2471 2472 2473 2474. Once those frames were receieved on the iPhone, it sends out correct ACKs for these in frames 2483 2484 2485 2486.
So the question is, why was the data originally sent to the iPhone in frams 2381-2384 not acknowleged? This based on as well as the fact that the iphone took forever to respond to those ACKs seems to me like there is some issue with packet reception on the WiFi interface. Note that both of these tests were carried out on 2.4GHz 802.11n in a relatively RF clean environment. But similar issues are seen at work where we have 5GHz deployed.
Frame Time From To Proto TCP-SEQ Len Description
2380 16.038190 64.15.114.78 10.101.2.100 TLSv1.2 181890 1510 Continuation Data
2381 16.039164 64.15.114.78 10.101.2.100 TLSv1.2 183334 1510 Continuation Data
2382 16.039284 64.15.114.78 10.101.2.100 TLSv1.2 184778 1510 Continuation Data
2383 16.039524 64.15.114.78 10.101.2.100 TLSv1.2 186222 1510 Continuation Data
2384 16.039610 64.15.114.78 10.101.2.100 TLSv1.2 187666 1510 Continuation Data
--snip--
2430 16.434535 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Window Update] 49388 > https [ACK] Seq=2767 Ack=183334
2431 16.434564 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#1] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482849 TSecr=82032821 SLE=189110 SRE=191974
2433 16.435119 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#2] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482851 TSecr=82032821 SLE=189110 SRE=193418
2434 16.440695 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#3] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482853 TSecr=82032821 SLE=189110 SRE=194862
2436 16.443284 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#4] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482858 TSecr=82032821 SLE=189110 SRE=194886
2437 16.443967 64.15.114.78 10.101.2.100 TLSv1.2 194886 1510 Continuation Data
2438 16.444295 64.15.114.78 10.101.2.100 TLSv1.2 196330 1510 Continuation Data
2439 16.444841 64.15.114.78 10.101.2.100 TLSv1.2 197774 1510 Continuation Data
2440 16.445598 64.15.114.78 10.101.2.100 TLSv1.2 199218 1510 Continuation Data
2441 16.446831 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#5] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482861 TSecr=82032821 SLE=189110 SRE=196330
2442 16.446943 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#6] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482861 TSecr=82032821 SLE=189110 SRE=197774
2443 16.446963 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#7] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482862 TSecr=82032821 SLE=189110 SRE=199218
2445 16.449042 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#8] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482863 TSecr=82032821 SLE=189110 SRE=200662
2447 16.468802 64.15.114.78 10.101.2.100 TLSv1.2 200662 1510 Continuation Data
2448 16.468898 64.15.114.78 10.101.2.100 TLSv1.2 202106 1510 Continuation Data
2449 16.469160 64.15.114.78 10.101.2.100 TLSv1.2 203550 1510 Continuation Data
2451 16.469222 64.15.114.78 10.101.2.100 TLSv1.2 204994 1510 Continuation Data
2453 16.471950 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#9] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482884 TSecr=82032821 SLE=189110 SRE=202106
2454 16.472080 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#10] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482885 TSecr=82032821 SLE=189110 SRE=203550
2455 16.472099 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#11] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482885 TSecr=82032821 SLE=189110 SRE=204994
2456 16.472656 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#12] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482885 TSecr=82032821 SLE=189110 SRE=206438
2457 16.482632 64.15.114.78 10.101.2.100 TLSv1.2 206438 1510 Continuation Data
2458 16.482730 64.15.114.78 10.101.2.100 TLSv1.2 207882 1510 Continuation Data
2459 16.482957 64.15.114.78 10.101.2.100 TLSv1.2 209326 1510 Continuation Data
2460 16.483060 64.15.114.78 10.101.2.100 TLSv1.2 210770 1510 Continuation Data
2462 16.485924 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#13] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482897 TSecr=82032821 SLE=189110 SRE=207882
2463 16.486004 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#14] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482898 TSecr=82032821 SLE=189110 SRE=209326
2466 16.487264 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#15] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482898 TSecr=82032821 SLE=189110 SRE=210770
2467 16.487287 10.101.2.100 64.15.114.78 TCP 2767 78 [TCP Dup ACK 2430#16] 49388 > https [ACK] Seq=2767 Ack=183334 Win=131072 Len=0 TSval=452482898 TSecr=82032821 SLE=189110 SRE=212214
2471 16.496460 64.15.114.78 10.101.2.100 TLSv1.2 183334 1510 [TCP Fast Retransmission] Continuation Data
2472 16.496793 64.15.114.78 10.101.2.100 TLSv1.2 184778 1510 [TCP Retransmission] Continuation Data
2473 16.500837 64.15.114.78 10.101.2.100 TLSv1.2 186222 1510 [TCP Retransmission] Continuation Data
2474 16.501164 64.15.114.78 10.101.2.100 TLSv1.2 187666 1510 [TCP Retransmission] Continuation Data
2475 16.504123 64.15.114.78 10.101.2.100 TLSv1.2 212214 1510 Continuation Data
2476 16.504189 64.15.114.78 10.101.2.100 TLSv1.2 213658 1510 Continuation Data
2477 16.504449 64.15.114.78 10.101.2.100 TLSv1.2 215102 1510 Continuation Data
2478 16.504515 64.15.114.78 10.101.2.100 TLSv1.2 216546 1510 Continuation Data
2483 16.505236 10.101.2.100 64.15.114.78 TCP 2767 78 49388 > https [ACK] Seq=2767 Ack=184778 Win=129600 Len=0 TSval=452482909 TSecr=82033236 SLE=189110 SRE=212214
2484 16.505256 10.101.2.100 64.15.114.78 TCP 2767 78 49388 > https [ACK] Seq=2767 Ack=186222 Win=129600 Len=0 TSval=452482910 TSecr=82033236 SLE=189110 SRE=212214
2485 16.508142 10.101.2.100 64.15.114.78 TCP 2767 78 49388 > https [ACK] Seq=2767 Ack=187666 Win=129600 Len=0 TSval=452482916 TSecr=82033236 SLE=189110 SRE=212214
2486 16.508168 10.101.2.100 64.15.114.78 TCP 2767 66 49388 > https [ACK] Seq=2767 Ack=212214 Win=106496 Len=0 TSval=452482917 TSecr=82033236
2487 16.508615 10.101.2.100 64.15.114.78 TCP 2767 66 49388 > https [ACK] Seq=2767 Ack=213658 Win=138624 Len=0 TSval=452482917 TSecr=82033270
2488 16.508635 10.101.2.100 64.15.114.78 TCP 2767 66 49388 > https [ACK] Seq=2767 Ack=216546 Win=135712 Len=0 TSval=452482917 TSecr=82033270