I am sorry that no one has answered this before.
Skype relies on what connections it can make to spread the load of video chats.
There is no central server other than the one that hold the names.
Essentially connections are supposed to be Peer to Peer.
As well as Little Snitch showing the app connects you to many end users to achieve some sort of balance you may also suffer from your ISP trying to "Manage" Peer-to-Peer file sharing under some "Fair Use" policy.
This may mean they slow down the connections they thinks are P2P file Sharing.
It may mean that in what they call "Busy" periods the speed is reduced (the polices rarely state when "Busy" actually is).
Please, if posting Logs, do not post any Log info after the line "Binary Images for iChat"
iMac 2.5Ghz 5i 2011 (Mountain Lion 10.8.2)
G4/1GhzDual MDD (Leopard 10.5.8)
MacBookPro 2Gb (Snow Leopard 10.6.8)
Mac OS X (10.6.8),
Couple of iPhones and an iPad
"Limit the Logs to the Bits above Binary Images." No, Seriously