Product bug: unknown unicast traffic storms from thunderbolt displays

Following up on my last post, I forgot to add in my case we see between 8.5-9.5K packets/second when this happens. Here is an excerpt from my wireshark trace:

I now have a case open with Apple on this.

All I can suggest to folks is if you have a large set of Thunderbolt displays at your company you're bound to see this sooner or later, so implement storm control if you haven't already.

Same problem here. It only happens when attached to the network through a thunderbolt display. Both MacBook Airs and MacBookPros have been observed doing this. It's been bad at times as our switches flood the traffic to all ports on the same VLAN (we are now sending 0000.0000.0000 to null0 on our access switches to minimize the impact). I can repeatably create the error condition every time by simply rebooting the attached MacBook.

We have experienced the same issue with increasing frequency as more Thunderbolt displays are introduced into our environment in the last year. On a gigabit port, the display has no problem generating 800mbit/s or more of traffic (~500kpps) - which is then flooded to every port in the same VLAN (~400 user ports in our case). For 100mbit/s users, this essentially floods them off the network.

Here is a detail I don't see mentioned above -- this happens even when a laptop/computer is not connected to the display. The first case we had of this happening was with a display that had no thunderbolt parent device attached. Shutting down the switchport and no-shutting it (bouncing the link on the display) resolves this until the next time it happens.

It looks like whatever crap resides in various buffers is used to construct the resulting Ethernet frames. I did not perform a packet capture this time, but the last time it happened the entire Ethernet header was null bytes with the body being mostly-null but the same random-looking noise in the rest of the frame. The frame was interpreted by Wireshark and others as a type of Fiber Channel, but I think that was just the default case that matched many of the null characteristics. The exact same frame was reflected in each packet sent (as opposed to each frame being different/randomized from the predecessor)

Well we've seen this too. I've actually been dealing with this for about a month and finally concluded it was the Thunderbolt monitor as opposed to a faulty switch port, etc. We've now seen it on at least two displays and had to implement storm control on our Ciscos to combat it. In wireshark we see:

Frame 678838: 90 bytes on wire (720 bits), 90 bytes captured (720 bits) on interface 0

Ethernet II, Src: IfmElect_00:00:00 (00:02:01:00:00:00), Dst: 00:00:00_00:01:b8 (00:00:00:00:01:b8)

Type: Unknown (0x4000)

Data (76 bytes)

Data: 0001333300000016a820664a180386dd6000000000240001...

The source, destination and data is always the same and I believe an exact match for what we saw on both Thunderbolt displays.

Question is, has anyone pursued this with Apple and is there an explanation and fix?

mclovinn, would you mind sharing your Apple case number? I am putting in a case for my company and would like to be able to provide them with proof that this is not just a one-off issue. The more examples they see of this issue, the more likely they are to actually look into the root cause.

Another note, I had talked to an Apple support member several weeks ago regarding the issue. They had suggested to make sure that the displays are updated to the latest firmware, 1.2. After that, I checked one of the displays that had experienced this issue. It did not have the newest firmware so I updated it but, just this morning, the same display had the issue again. So the latest firmware patch is not a fix for this problem.

We continue to have this issue - this morning we had our 4th unique thunderbolt (but each has been responsible for numerous instances) display knock major parts of our network offline by sending a huge stream of junk traffic (which was treated as unknown unicast).

We will continue to apply pressure on Apple Support and our account rep regarding this; it would be beneficial if anyone else experiencing this did the same and shared their case numbers

Was anyone able to get any info from Apple about this? We've had a very similar issue with 2 Thunderbolt displays (that we know of) and I was hoping to get some more information.

FWIW, we are seeing a different source MAC address(6000.0000.0038) and have not had a chance to capture it in Wireshark yet.

Out of curiosity, has anyone noticed any seeming correlation with Yosemite? Or, put another way, have folks seen this problem with monitors that have been connected to Macs running on 10.9 or below? I realize it seems to be primarily a display issue, but just wondering.

Thanks!

PS, we've seen this issue once in my place of work. Took down a big chunk of the network; so we unplugged the display from the network. NetOps reported excessive multicast traffic from the laptop/display pair a couple of weeks before the unicast storm; so we had been looking at discoveryd. But the unicast stuff was really malformed so we kind of went in another direction. But I note Apple got rid of discoveryd in 10.10.4 so it made me curious about it again.

We had this issue again earlier this week so I contacted Apple support and they said that they "believe" the issue is actually in the OS on the MAC not an actual issue with the display. They suggested upgrading to 10.11 (El Capitan) and believe the fix is in the OS. If any of you out there experience this issue on El Capitan, please post it here and contact support so they are aware.

To add additional information;

We have found that as stated in the original post, using a thunderbolt to ethernet adapter solves the issue. For about 4 months we only had 1 display causing trouble. We connected a thunderbolt to ethernet adapter to the monitor and haven't had further trouble. We had a different display exhibit the same trouble last week. We also moved him to an adapter and haven't seen the issue. I'll report back when I have more information but I don't believe I'll see the trouble again. I suppose this doesn't rule out the OS as the issue since it could be a bug with 10.10 and a thunderbolt display. We aren't ready to allow our users to upgrade to 10.11 yet. When we do, I'll remove the adapters and see if the trouble comes back. My hunch is that this has nothing to do with the OS and is a batch of faulty NIC hardware in the displays.

We started seeing this in mid to late 2014. There would be a traffic flood of around 700,000 pps, to and from a MAC address starting with 6000. Each time we tracked down the offender, it would be a Thunderbolt running Yosemite. Very odd thing is it would always happen on Monday early afternoon, say around 1:30.

A good work-around from the network side if you're running Cisco switches is the Storm Control feature. Since this is unicast traffic, something like "storm-control unicast level pps 200k" on all desktop ports should catch it in the act.

I'm somewhat skeptical Apple actually identified and fixed this in El Capitan. This is essentially a network denial of service attack somehow tied to hardware, and really should have been fixed openly via a driver update. Given problems we've had with El Capitan related to Office, we won't be upgrading anytime soon.

In case anyone is interested, there is a case opened with Apple and this issue is currently being investigated. Case Number #833098033

Any update on this? We just had a TB display kicked off our network by our network ops staff for exhibiting this problem.