My network connection completely died - "failed: 49 - Can't assign requested address"

Same situation on my host. I hope 10.9.2 already fix this issue.

➜ ~ uname -a

Darwin ksn135imac27.local 13.1.0 Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64

➜ ~ uptime

8:16 up 20 days, 21:34, 2 users, load averages: 1.42 1.46 1.33

➜ ~ netstat -an | egrep "^tcp|^udp" | egrep -c "FIN_WAIT_1|LAST_ACK"

2

➜ ~ netstat -an | egrep "^tcp|^udp" | egrep -vc "FIN_WAIT_1|LAST_ACK"

192

➜ ~

Unfortunately it looks like 10.9.2 hasn't fixed this. At least for me.

I noticed it a while ago on 10.9.1 and then thought it had been fixed.

Then today I've been experiencing weird things. Parts of web apps failing, attempts to send files by sftp failing to even start, so I tried running that that netstat command from earlier in the thread...

$ netstat -n | egrep '^(tcp|udp)' | awk '{print $NF}' | sort | uniq -c

76 *.*

3 CLOSE_WAIT

153 CLOSING

15 ESTABLISHED

11018 FIN_WAIT_1

44 FIN_WAIT_2

4190 LAST_ACK

2 SYN_RCVD

2 SYN_SENT

😟

Machine running 10.9.2 started acting funny today...

Sure enough:

homer:~ georgn$ uptime

12:38 up 27 days, 19:18, 4 users, load averages: 1.92 1.50 1.30

homer:~ georgn$ netstat -an | egrep -c 'ACK|WAIT'

103

homer:~ georgn$ uname -a

Darwin homer.bitmover.com 13.1.0 Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64

Machine was under a little bit of memory pressure (running vmware, etc). Not sure if there's any conclusion to be drawn... that said, killing apps does not remediate anything.

In my case, the uptime was 59 days with everything working fine. But I hit the limit today, and no network connections were possible after that. So my guess is that this problem is not strictly related to the duration of uptime, rather the amount of network activity.

I can confirm someone at Apple is working on this. My bug report has just been closed as a duplicate of another one.

This is a typicall Apple "trick" to avoid to let us know the progress of a bug report as we don't have access to the "parent" record.

So, let's cross fingers for 10.9.3 :-)

Seb

I could be wrong but I believe Apple has already closed duplicate tickets for other people in this thread as well. Many have filed tickets. Hopefully I'm totally wrong and Apple is working on it.

Don't know if they are working on it.....but Apple closed my ticket tonight as well.

-Kevin

Ok guys, hope I am not violating any agreement with Apple... my bug report is still alive, bug id

16336220.

I can not see this issue (yet), even though the numbers are very bad here:

$ netstat -an | egrep "^tcp|^udp" | egrep -c "FIN_WAIT_1|LAST_ACK" ; netstat -an | egrep "^tcp|^udp" | egrep -vc "FIN_WAIT_1|LAST_ACK"

5830

299

Uptime is over 37 days, so this is not uptime-related.

If you hit this issue again, please do what Apple folks suggested in my report:

"1. Login as an admin user

2. Launch Terminal application

3. To become "root", enter the following command:

sudo -s

4. Enter your password at the prompt

5. Run the get-mobility-info command:

/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/Resources/g et-mobility-info

6. Once the command is finished the command print out where the result has been collected, something like Network data collected to /tmp/mobility-config-1234.tar.gz (Note: the number in the file name is going to be different)

7. Please upload this file to your bug report"

For step 7., just share the file in here or PM me and I will attach it to my report.

I didn't know it until after the reboot, so I didn't get any screenshots, but I had this problem today too. I am on 10.9.2, was up for about 16 days. *****.

Another update from Apple engineers:

"If you see the problem again, please provide

1. get-mobility-info

2. sysdiagnose

3. Run tcpdump for 5 minutes and upload the packet capture. This will not collect any of the user's data, it will mainly collect the packet headers.

$ tcpdump -i <interface name> -s 100 -w tcpdump.pcap

4. Run the following command and upload the output:

$ sudo timerfires -n kernel_task -t 300 > /var/tmp/timers.txt

Please provide your response or results by updating your bug report.

Please compress any bundled files (e.g. nested folders) prior to uploading."

The numbers are getting worse every day here, but not there yet:

$ netstat -an | egrep "^tcp|^udp" | egrep -c "FIN_WAIT_1|LAST_ACK" ; netstat -an | egrep "^tcp|^udp" | egrep -vc "FIN_WAIT_1|LAST_ACK

"

7321

1221

$ uptime

10:37 up 42 days, 11:42, 4 users, load averages: 1.50 1.92 2.15

This is happening to me as well. On multiple machines (mac mini, MacBook Pro 13", a Mavericks VM in VMware). It has happened in all versions of 10.9 that I've run (.0,.1, .2).

Got this on my iPhone 5 iOS 7.0.4 about a day ago.

rts-iPhone:~ root# uptime

06:29am up 51 days 6:19, 1 user, load average: 2.02, 1.00, 0.81

rts-iPhone:~ root# netstat -an | egrep "^tcp|^udp" | egrep -c "FIN_WAIT_1|LAST_ACK"

392

rts-iPhone:~ root# netstat -an | egrep "^tcp|^udp" | egrep -vc "FIN_WAIT_1|LAST_ACK"

16630

rts-iPhone:~ root# ping apple.com

PING apple.com (17.149.160.49): 56 data bytes

--- apple.com ping statistics ---

4 packets transmitted, 0 packets received, 100% packet loss

rts-iPhone:~ root# ping umanitoba.ca

PING umanitoba.ca (130.179.16.50): 56 data bytes

64 bytes from 130.179.16.50: icmp_seq=0 ttl=46 time=409.726 ms

64 bytes from 130.179.16.50: icmp_seq=1 ttl=46 time=431.648 ms

64 bytes from 130.179.16.50: icmp_seq=2 ttl=46 time=454.530 ms

64 bytes from 130.179.16.50: icmp_seq=3 ttl=46 time=478.615 ms

--- umanitoba.ca ping statistics ---

4 packets transmitted, 4 packets received, 0% packet loss

round-trip min/avg/max/stddev = 409.726/443.630/478.615/25.670 ms

Somehow I am not surprised that iOS is also affected. Both iOS and OSX share the XNU kernel.

I just hit the issue again with two different machines running 10.9.2. One machine hit it without the LAST_ACK/FIN_WAIT_1 connections in netstat. Not sure what is going on there. The other machine had the characteristic large number of sockets in those state.

I wish Apple exposed the changesets between 10.8 and 10.9. This would potentially be fixed much faster if there were more eyes on the code.

I had this exact issue with ever-growing FIN_WAIT_1 connections on a Mac OS 10.9.2 server.

Using:

netstat -n | egrep '^(tcp|udp)' | grep FIN_WAIT_1

I was able to determine that *all* of the FIN_WAIT_1 connections were going to a destination port of 2195, which is Apple's Push Notification services. Once I created a firewall rule (on our gateway firewall) to block traffic from our server to anything on port 2195, the FIN_WAIT_1 connections stopped incrementing and the server has been stable.

It's still too early if this has resolved the overall connection issues - it's only been 3 days - but I thought that this might be something that others could look for on their systems to see if APN connections are at the source of their issues as well.

Of course, this is not a long term solution for some - you'll need to have APN services running for various services in Mavericks server. However, in our case, I'd much rather run without APN services than with a server that can't stay standing for more than 3 days at a time.

Thanks to everyone else who is trying to troubleshoot this issue. I wouldn't have gotten anywhere without your 9 pages of tips...

oh_techsuperpowers.com wrote:

I had this exact issue with ever-growing FIN_WAIT_1 connections on a Mac OS 10.9.2 server.

Using:

netstat -n | egrep '^(tcp|udp)' | grep FIN_WAIT_1

I was able to determine that *all* of the FIN_WAIT_1 connections were going to a destination port of 2195, which is Apple's Push Notification services. Once I created a firewall rule (on our gateway firewall) to block traffic from our server to anything on port 2195, the FIN_WAIT_1 connections stopped incrementing and the server has been stable.

This worked for me, was getting the network crash every 2-3 days on OS X server 10.9.2. I hope this is fixed in 10.9.3

Thanks.