Your WiFI download speed should be about 20 mp/s, you do have a serious issue.
My advice is:
1: update your routers firmware
2: run a wifi sniffer to find out what channels other routers are on in the area and set yours way from theirs
3: follow the advice here and change all your passwords etc.
4: if the router is still slow, consider replacing it, move closer to the router
Interesting that you found it to be the cause on your router. I just checked and on mine, QoS is on for both IPv4 and IPv6, and I get 100% of my ISP's speed, so it might not always be the cause. Sounds like it might depend on the type of router someone is using.
But if turning it off works in your case, that's good to hear.
Actually you should get the same bandwidth but higher latency... speedtest.net only test your internet connection.
ethernet = 1000 or 100 Mbits
Wifi (802.11g) = 54Mbits max
Your internet connection is much slower than ether of those, the difference with wifi however is that you can get interferance like the other posters here are suggesting. whichever is the common denominator is what speedtest will measure. in the case of ethernet it's showing your true internet connection throughput. in the case of WiFi it's showing your current wifi speed.
Just wanted to clarify.
So yeah... you need to see what channels other people are using. and use the most distant channel (consider that wifi channels over lap each other, i.e. 6 overlaps 7 and 5).
you might also want to try switching between g and n mode, as each uses different spectrums which will have different levels of interfearance depending on usage and location... you dont need the full bandwidth of n if you internet connection is only 20Mbits... becase g is twice that anyway.
I also had the same problem. Initally my speed was 3.8Mbps. I have a Macbook Pro running Mac OSX 10.6.8. My router is a Dlink DIR-655 with QoS. I turned off the QoS and my speeds returned. I ran Speedtest.net and recieved 15.8Mbps. That is what I was suppose to get. Then I went back into my router and turned QoS back on and rebooted the router. I performed another speedtest and I recieved similar results 16.5Mbps. So you were correct in the QoS causing the problem. But why it was happending I don't know must be some sort of conflict with the QoS processing. One last piece of info this started to happen after I changed my DNS to OpenDNS. Not saying that was the cause but it could have been. Now everything is working perfectly. Maybe someone else out there had the same issue. Goodluck!
Since IPv6 was mentioned, if your ISP is not using/deploying it right now, consider disabling it on the Mac. Many routers freak out when a device starts hammering them for its global prefix. My 2Wire napkin holder wouldn't let me download a single patch from Apple till I shut IPv6 down for now.