You may want to check if the adjacent networks are using the same WiFi channel you are using.
Try a utility such as iStumbler.
For 2.4GHz frequences, you want to be on a channel all by yourself and 5 channels away from the strongest signals. That means channels 1, 6, or 11 typically. Outside the US, some countries allow channels 12 & 13, so you may play different 5 channel apart games.
The 5GHz frequencies are better spaced so as long as you are not using the same channel, all is OK.
If you are using channels that conflict, then you can get interference that may slow down your WiFi.
Another approach is to make an Ethernet connection instead of WiFi, and see if your performance improves. If it does, then it may be WiFi interference. If it does not improve, then it may be how fast your Internet connection is working.
I know a lot of people using Cable based broadband that see performance slow down when school lets out and the neighborhood kids start watching YouTube videos and chewing up the shared neighborhood bandwidth.