how do I tune TCP SYN resend/retransmit rate for satellite?
On a satellite internet connection MacOSX (and, I believe, iPhones) have excessively aggressive SYN resend timing.
The existing settings make iTunes and SSL based services (banking, email etc) unreliable and often unworkable.
Plain HTTP (most web browsing) gives the appearance of working because browsers quietly retry.
I have been unable to find a setting in Preferences or sysctl to adjust this. And of course there's nothing useful on an iPhone. It would be helpful if there was some switch for "satellite" or "high latency" networks.
Summary:
When combined with the (common) TCP accelleration in a satellite modem, TCP connections from a Mac over satellite often get reset right at the start when the connection is commencing because of a race condition between the modem's stateful connection proxying and MacOSX's excessively rapid SYN resends.
Workaround:
Running a proxy between the Mac and the modem (in our case, in our firewall) where the proxy runs on a more conventional machine with much slower (normal) SYN resend times seems to work around the issue by making the Mac's connection to the proxy, and the proxy's connection over the satellite connection. Because the proxy's host has better TCP behaviour, the connections succeed.
Detailed problem description and explaination:
At times, SSL connections (HTTPS, POP3S, IMAPS) and other connections receive a batch of RST packets just after setup, leading to connection refused or connection reset by peer. Depending on the app, you get an assortment of errors messages of varying vagueness. iTunes is especailly useless here, just saying an unknown error occurred.
For something that reports the OS-level error you get message like this:
socket error ([Errno 54] Connection reset by peer)
Generally, if the connection succeeds it will stay up. The failure occurs at connection setup time.
From running tcpdumps on my firewall when this is happening, the behaviour is as follows:
For background to understand what follows: a TCP connection is established by you sending a SYN (synchronise) packet to the far end. The far end replies with a SYN/ACK acknowledging the connection. You then send an ACK ackowledging receipt of the SYN/ACK. And then you send and receive data packets.
A successful connection looks like this (you may need to make your window rather wide):
16:47:47.896426 172.16.3.7.50142 > X.X.X..995: S 2233798023:2233798023(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306458055 0,sackOK,eol> (DF)
16:47:48.597903 172.16.3.7.50142 > X.X.X.X.995: S 2233798023:2233798023(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306458755 0,sackOK,eol> (DF)
16:47:48.643500 X.X.X.X.995 > 172.16.3.7.50142: S 0:0(0) ack 2233798024 win 13312 <mss 1321>
16:47:48.644794 172.16.3.7.50142 > X.X.X.X.995: . ack 1 win 65535 (DF)
16:47:48.645639 172.16.3.7.50142 > X.X.X.X.995: P 1:307(306) ack 1 win 65535 (DF)
So: the normal SYN, a duplicate SYN 701ms later (much too early), a SYN/ACK from the far end and an ACK for that from this end. And then normal traffic.
An unsuccessful connection looks like this:
16:48:11.094536 172.16.3.7.50144 > X.X.X.X.995: S 2181400205:2181400205(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306481128 0,sackOK,eol> (DF)
16:48:11.797141 172.16.3.7.50144 > X.X.X.X.995: S 2181400205:2181400205(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306481827 0,sackOK,eol> (DF)
16:48:12.099393 172.16.3.7.50144 > X.X.X.X.995: S 2181400205:2181400205(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306482127 0,sackOK,eol> (DF)
16:48:12.400934 172.16.3.7.50144 > X.X.X.X.995: S 2181400205:2181400205(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306482427 0,sackOK,eol> (DF)
16:48:12.702356 172.16.3.7.50144 > X.X.X.X.995: S 2181400205:2181400205(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306482727 0,sackOK,eol> (DF)
16:48:13.003581 X.X.X.X.995 > 172.16.3.7.50144: S 0:0(0) ack 2181400206 win 13312 <mss 1321>
16:48:13.005832 172.16.3.7.50144 > X.X.X.X.995: S 2181400205:2181400205(0) win 65535 <mss 1460,nop,wscale 4,nop,nop,timestamp 1306483028 0,sackOK,eol> (DF)
16:48:13.006593 X.X.X.X.995 > 172.16.3.7.50144: R 1:1(0) win 0
16:48:13.007379 172.16.3.7.50144 > X.X.X.X.995: . ack 1 win 65535 (DF)
16:48:13.007778 172.16.3.7.50144 > X.X.X.X.995: P 1:307(306) ack 1 win 65535 (DF)
16:48:13.008437 X.X.X.X.995 > 172.16.3.7.50144: R 1:1(0) win 0
16:48:13.009220 X.X.X.X.995 > 172.16.3.7.50144: R 1:1(0) win 0
A normal SYN, which gets repeated 703ms later, then 302ms after that, then 301ms later, then 302ms later. Then a SYN/ACK from the far end 2097 milliseconds after the first SYN. Then yet another repeated SYN from our end 303ms after the last SYN, then immediately an RST. Our ACK follows (probably already dispatched by the Mac before the RST, but arriving at the firewall just after the RST) and a PUSH with the first data. Two more immediate RSTs come back, presumably for each of these data packets.
By comparison, our BSD firewall resends the first SYN 6 seconds after the original, then even slower. Comparsed to 600-700ms, then every 300ms from the Mac. The Mac's resends interact very badly with sat modem TCP acceleration and sat latency.
There seem to be a few things going on here:
Firstly, something is sending too many SYN packets. There is no need to send repeated SYNs every 300ms. Certainly the best round trip latency over a satellite is about 700ms. The coincidence of the satellite latency and the first resend time probably exacerbates things.
Secondly, sending a SYN _after_ the far end's SYN/ACK has come back seems to elicit an RST packet. I think this is because the modem has established state for the connection because it has seen the far end's reply, and is rejecting what it now thinks is an attempt to set up a new connection using the same ports (forbidden).
Third, this happens more at times of congestion, because the SYN/ACK takes a little longer to come back.
Fourth, the immediacy of the RST packets seems to suggest the RSTs are coming from the modem. So it is probably doing TCP accelleration, and that is producing the RSTs.
There are two problems here:
Up front, the Mac is producing SYNs too often when starting a connection. This exacerbates the problem into something that happens frequently (when it happens at all). This may be TCP "fast mode", but regardless I cannot seem to turn it off.
Second, one could argue that that modem is being overly strict about rejecting the extra SYN. Probably better to pass it through to the far end and let that decide whether to RST it. The modem is doing TCP accelleration (which is fine – it permits better behaviour once the connection is established) but is considering the connection "established" too early. It seems to do it on seeing the SYN/ACK, instead of waiting to see the third packet (the ACK).
I also suspect this happens with plain HTTP as well, but that browsers hide it by retrying connections. The modem is not going to get fixed.
While I think I'm working around this in my firewall now by diverting all outbound satellite connections through a proxy, many users will not have that option, nor any ability to diagnose the issue. They'll just see "Macs don't work on satellite". I'm not alone; there is a scattering of posts on Whirlpool recounting similar symptoms on satellite with Macs.