I had this exact same problem. It suddenly started when I reset my router to the original settings. Every thing suddenly was slow, then I finally figured out it was Mac Mail, searched Google for an answer, found this thread, then went in to my router page, since no one could figure out the cause. ***BINGO***
I fixed it by going to the Firewall settings.
IP Flood Detection [square box here] Enable
I un-checked the square box and slow speed be gone!!!!!! This was in the Ubee router at
http://192.168.100.1/RgContentFilter.asp
but for other routers, it might also have a Firewall setting like this some where. Hunt through your settings at
http://192.168.100.1/
is what gave me the idea to go hunting through the router, a uBee router. Thanks.